From raw audio to TFRecords to data streaming

From audio to TFRecord to a readily-usable dataset

With the newest iteration of its custom M1 chip, the M1 Pro and M1 Max versions, Apple has given the Machine Learning community a powerful tool. To some extent, however, this power can only be unleashed if the system is set up correctly.

Without a doubt, the ImageNet dataset has been a critical factor in developing advanced Machine Learning algorithms. Its sheer size and a large number of classes have been challenging to handle.

Custom training loops offer great flexibility. You can quickly add new functionality and gain deep insight into how your algorithm works under the hood. However, setting up custom algorithms over and over is tedious. The general layout often is the same; it’s only tiny parts that change.

After you have finally created that training script it’s time to scale things up. From a local development environment, be it an IDE or Colab, to a large computer cluster, it’s quite a stretch. The following best practices make this transition easier.

Have you ever had a dataset, and asked: Does this model learn something different from that model?