Problem Approach

• Start small. Experiment and iterate quickly on smaller models and datasets before scaling up.
• Keep a spreadsheet of your experiments of things that you tried
• Do Kaggle comps. Definitely one of the best ways to learn. Read the forums and discussions. Listen to what others are doing.
• Lookup research papers related to the problem you are trying to solve. Chances are they may have come across a solution and approach that will be relevant.
• Before diving into the largest biggest model, start with something appropriate that fits your dataset.

  • How to pick the right pretrained model

    https://forums.fast.ai/t/how-to-pick-the-right-pretrained-model/8481

    • You always want to go for the smallest model that works well for your data. Up until earlier this year, people usually start with VGG16 or VGG19, but Resnet is also a great choice for fine tuning. Start with Resnet18, then to Resnet34 and Resnet50. You could also try the newer models in ResNext or Nascent nets.
    • Bigger models is not always better. They usually overfit your train data. What you really care about is Validation loss. Keeping it small, yet closer to training loss is tricky and may need some regularization like DropOut or WeightDecay etc. FastAI provides easy access to all of them.
    • Whether you need more data or not depends on the task. If you are trying to identify Hotdog or not-hotdog you may not need lots of data. But if you are trying to read Street Signs and Logo from your pictures, you may need more data.
    • tldr: When working on a new problem, always start with simple networks - Resnet18 / 34 or 50 are good choices.
• When trying to find baselines, a good default for a learning rate tends to be 3e-3. Try that as a starting learning rate, especially when benchmarking basics such as image processing changes.

Data

• Know your data well. Look at it manually, understand its distributions.
• Pick out imbalances. Do you need to oversample / under sample?
• Image Cleaning
  • Sometimes the best way to get the most our of your models is to improve the input. Try various ways to improve image quality for your model.
  • Try image enhancements that normalise well across the datasets (like consistent cropping) or play with lighting and contrast settings that make it easier.
  • Normalise exposures
  • Explore the effects of single-ing out image RGB channels

Hyper-parameters, training and models.

• Determine if your problem is a classification or regression problem. ie If regression loss functions like MSE are probably more relevant. For classification you may want to look at CrossEntropyLoss
• Use Transfer learning and pre-trained weights when you can.
• Fit one cycle is great.