So you understand you performance problems and need to do something about them. Your measurement values suggest the issues are in your own code or design. Now you are considering your options.

#Introduction [2019-10-08] Disclaimer: This is a draft

You look at the code base and it sort of stinks. The organic growth is obvious and no one ever cared about the architecture. Technical Debt is a mountain higher than everest. Your first thought is to toss it all out and rewrite…

You are most likely wrong.

If suggesting to your boss and/or rest of the management that you want to rewrite the whole thing you should consider what they will hear. Mangement will hear that you intend to take 30-50% of the developers (you want all of them) and stick them to a project with very uncertain outcome for quite a long time. This will basically only cost a fortune and you can’t give any credible evidence that your will succeed (you can’t really give that). Any sane manager will likely turn this down.

Instead I say focus your energy on picewise refactoring. Using the measurement data it’s now possible to identify the most impacting functions for a refactor.

#Prerequisits Even if not 100% necessary the following will ease your pain considerably

• Automatic build on changes to specific branches
• Automatic deployment of successfull builds
• Automatic execution of post-deployment scripts/actions

If you don’t have this you should really consider your build/deploy pipeline first.

#Refactoring Tackle one problem at the time. Your attention should be fullstack. The server API is the interface to the client. Try to keep the API intact - this will avoid opening too many potholes at once.

1. Write simple integration tests (be it stand alone CLI tool or with some framework)
2. Collect data with nothing changed - this is your baseline
3. Implement fine-grained instrumentation at some strategic points (DB latency, Business Logic, etc..)
4. Hide the old code behind an interface
5. Create a new instance of the interface for the new code
6. Start implementation of the new interface (at the start the new instance should have 0 latency)
7. Bridge to the “old” codebase through interfaces and replace them (as you go along) with new refactored code
8. Swap out as necessary
9. Constantly run the integration test against the new and old code
10. Repeat to 6 until done

The good thing with this approach is that you can implement improvements picewise. Your fine grained instrumentation will tell you in which area of the code base you need to start. Could be the DB could be the business logic or something else.

Make sure:

• For EVERY interface you implement make sure you add fine-grained instrumentation (before and after)
• You always run the integration tests, it should be simple and allow for automation
• Constantly compare integration test results to the base line (note: MUST BE FROM SAME ENVIORNMENT!)

Example

This is more like pseudo code but it should give the idea of the stages you will go through.

Before:

    void code_we_are_working_on(...) {
        // do stuff
        x := Instrumentation.Start("Some Identifier");
        some_func(...);
        Instrumentation.End(x);
        // do more stuff
        return some_value;
    }
    void some_old_func(...) {
        // do magic stuff
    }

Preparation:

    interface functionality {
        void some_func(...);
    };

    class class_old_code_base : functionality {
        void some_func(...);
    }

    functionality code = new class_old_code_base();
    static void code_we_are_working_on(...) {
        // do stuff
        x := Instrumentation.Start("Some Identifier");
        code.some_func(...);
        Instrumentation.End(x);
        return some_value;
    }
    
    void class_old_code_base::some_func(...) {
        // do magic stuff
    }

Refactor:

    interface functionality {
        void some_func(...);
    };
    class class_new_shiny_code_base : functionality {
        void some_func(...);
    }
    // You can swap this out based on command-line and/org preprocessor..
    functionality code = new class_new_shiny_code_base();
    static void code_we_are_working_on(...) {
        // do stuff
        x := Instrumentation.Start("Some Identifier");
        code.some_func(...);
        Instrumentation.End(x);
        return some_value;
    }
    void class_new_shiny_code_base::some_func(...) {
        // do magic stuff
    }

Obviously you can do this all manually but I would assume that anyone maintaining a larger system has build automation.

#Build Automation The intent here is to discuss development - not production. For production CI/CD use a proper tool. There are several to choose from (Jenkis, Pupper, Chef and friends) but here is a very simple setup if you run git/gitlab/github.

1. Make sure the builds can run from the command line
2. Create a web-hook postback for your source repository
3. Listen for that postback and invoke your build script when it’s called

The build script should do an update of the branch in question, build it and deploy it. You may leave out DB changes to be handled differently.

I’ve mantained quite large projects with this methodology as a baseline and it works fine. I wouldn’t use it in a production environment but for development it’s fine.