SQL Tokenization function?

[virendrakabra14]

I'm trying to tokenize a SQL.

For example

select table1.`col1 with spaces`, table2.col2 from table1 join table2 order by table2.col2

to

['select', 'table1.`col1 with spaces`', 'table2.col2', 'from', 'table1', 'join', 'table2', 'order', 'by', 'table2.col2']

or something similar.

Can someone point to in-built functions for doing this?

I'm guessing it should be dialect-specific (e.g. sqlglot/dialects/mysql.py).

[tobymao]

sqlglot/sqlglot/dialects/dialect.py

Line 506 in 706fac3

def tokenize(self, sql: str) -> t.List[Token]:

check out the tokenizer