Sagor Sarker

Large text data token counting fast

April 28, 2024 less than 1 minute read

In this blog I will share codes to count tokens from large dataset fast.

April 4, 2024 1 minute read

In this blog I will note down some tips to process huggingface datasets

January 17, 2024 1 minute read

In this blog I will share how to process large CSV file using pandas

January 15, 2024 less than 1 minute read

In this blog I will share how to do multiprocessing on large CSV file with iterator

January 11, 2024 less than 1 minute read

In this blog I will share how to do simple multiprocessing in python over pandas dataframe