Cobra: How can I include Cobra in my actual project structure ?

[fallais]

I have this structure (not exhaustive) :

models/
services/
cmd/
  server/
    routes/
    Dockerfile
    main.go
  collector/
    Dockerfile
    main.go

server and collector are my two micro-services. They both have a Dockefile. That means I need to create two images at each build. Boring. That is why I want to migrate to Cobra.

But it seems that my vision of cmd interferes with the Cobra's one.

How can I do please ? I think about putting a main.go at the root of cmd/. But how can I link the two commands to my two existing main.go ?

Then, the Dockerfile, I assume that it will also be placed at the root of cmd/ ?

In fact, problem is that my cmd does not contain commands, but the whole micro-services.

[jharshman]

@fallais I think I need a bit more clarity on your question.
What exactly do you want to utilize Cobra for?

[fallais]

@jharshman : Everytime I want to build the two Docker images that correspond to my two microservices (server and collector). I need to build two images, from two Dockerfile. Inside each container, there is an executable, so I need to go build two times.

I would like to go into a continuous integration and build only one container each time I push on my master branch.

To do so, I need to package my two microservices into one. I have been hearing about Cobra to acheive that.

Then I could add into the Dockerfile only on executable. And play with : mysoftware server --option1 --option2 and mysoftware collector--option1 --option2. The software role will be defined at the startup.

Hope it is clear enough.

[umarcor]

@fallais, thanks for the clarification. Yes, cobra is suitable for your use case. However, let's ignore the Dockerfiles for now; it is not relevant for the golang codebase, and it adds unnecessary complexity to the question. I would rephrase the question as:

I have two independent golang apps which are built to separate binaries. Now, I want to integrate/merge both of them in a single binary. How can I use cobra to achieve it?

I suggest to start by creating a completely new app structure, using cobra's generator:

go get -u github.com/spf13/cobra/cobra
mkdir prjdir
cd prjdir
cobra init --pkg-name github.com/user/prjdir
cobra add server
cobra add collector
go mod init github.com/user/prjdir

You will get the following structure:

prjdir/
LICENSE
main.go
cmd/
  root.go
  collector.go
  server.go

Now, check the codebase, and play with it:

root@e05d1fadee50:/go/prjdir# go run main.go
A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

Usage:
  prjdir [command]

Available Commands:
  collector   A brief description of your command
  help        Help about any command
  server      A brief description of your command

Flags:
      --config string   config file (default is $HOME/.prjdir.yaml)
  -h, --help            help for prjdir
  -t, --toggle          Help message for toggle

Use "prjdir [command] --help" for more information about a command.

root@e05d1fadee50:/go/prjdir# go run main.go server
server called

root@e05d1fadee50:/go/prjdir# go run main.go collector
collector called

For your specific use case, I would:

• Merge root.go into main.go.
• Move prjdir/cmd/* to prjdir/. Adapt package names accordingly.

You will get:

prjdir/
LICENSE
main.go
collector.go
server.go

Now you can merge it with your actual structure:

cmd/
LICENSE
main.go
collector.go
server.go
  server/
    routes/
    main.go
  collector/
    main.go

where:

• server.go will import github.com/user/prjdir/server and collector.go will import github.com/user/prjdir/collector.

The main point you need to take into account is that a binary must have a single main function and that main packages cannot be imported. Hence, prjdir/cmd/server/main.go and prjdir/cmd/collector/main.go cannot be package main; you need to rewrite them to e.g. package server and package collector.

Should you want to preserve the possibility to build each of the services as independent binaries, I suggest to add a subdir to each of them. I.e.:

cmd/
LICENSE
main.go
collector.go
server.go
  server/
    routes/
    main/
      main.go
    lib.go
  collector/
    main/
      main.go
    lib.go

In this context, server.go would import server/lib.go, but not server/main/main.go (the same applies to collector):

• go get github.com/user/prjdir would build/install the cobra app, which includes both services.
• go get github.com/user/prjdir/server/main would build/install the server service.
• go get github.com/user/prjdir/collector/main would build/install the collector service.

However, this is up to you. If you don't want to preserve the possibility for independent builds, nor the services being imported by third-party projects, the following structure might be easier to maintain:

cmd/
LICENSE
main.go
  server/
    routes/
    main.go [package != main, includes the codebase of server.go above]
  collector/
    main.go [package != main, includes the codebase of collector.go above]

The same applies to the Dockerfiles. If services are self-contained, it might be worth creating a single image. However, if they need different dependencies, it might still be useful to create two different images, even though you build a single binary.

EDIT

If you feel that https://github.com/spf13/cobra/tree/master/cobra falls short of explaining how it can be useful in cases as yours, please do not hesitate to propose a PR!

[fallais]

Thanks a lot for this really complete answer !! I can't wait to test this ! :)

[jharshman]

@fallais , sounds like you got your questions answered. I'm going to go ahead and close this issue. If you have any more follow up questions, feel free to reach out again!

[fallais]

@jharshman Yes ! Thanks a lot again :)

[fallais]

@jharshman : To be honest, I do have a question. If I understand correctly, the whole code that I had it my main.go of each microservices goes into the Run() of each Cobra commands ? I do not find this beautiful. Is there another way to do this ?

Thanks a lot

[umarcor]

If I understand correctly, the whole code that I had it my main.go of each microservices goes into the Run() of each Cobra commands ? I do not find this beautiful. Is there another way to do this ?

Just put the code of each microservice in a different function, and call those functions from Run(). It is up to you to define these functions on a single package or in a separate package. A structure I use commonly is:

• root/:
  
  
  
  • lib/: business logic
  
  
  • cmd/: entrypoints to business logic
  
  

This allows users to use your project as a CLI tool or as a library.

[fallais]

But, the actual main I have is quite huge.. I mean I am declaring the DAOs, the services, the routes... I would have to add all of this code into the Run() of the Cobra command ?

If you look above, you will see my actual project structure. One main.go for each microservices.

[fallais]

For example, this is the main.go of the collector microservice.

package main

import (
    "flag"
    "net"
    "net/http"
    "time"

    "myProject/cmd/collector/runner"
    "myProject/cmd/collector/shared"
    "myProject/cmd/collector/system"
    "myProject/dao/mongodb"
    "myProject/services"

    "github.com/prometheus/client_golang/prometheus/promhttp"
    "github.com/robfig/cron"
    "github.com/sirupsen/logrus"
    "github.com/zenazn/goji/graceful"
)

var (
    bindAddress       = flag.String("bind", ":8000", "Network address used to bind")
    logging           = flag.String("logging", "info", "Logging level")
    configurationFile = flag.String("configuration_file", "configuration.yml", "Configuration file")
    databaseHosts     = flag.String("db_hosts", "localhost:27017", "Database hosts")
    databaseNamespace = flag.String("db_namespace", "xxxxx", "Select the database")
    databaseUser      = flag.String("db_user", "", "Database user")
    databasePassword  = flag.String("db_password", "", "Database user password")
    runOnStartup      = flag.Bool("run_on_startup", false, "Run on startup ?")
    recycleOffset     = flag.Duration("recycle_offset", 240*time.Hour, "Recycle offset")
)

func init() {
    // Parse the flags
    flag.Parse()

    // Set the logging level
    switch *logging {
    case "debug":
        logrus.SetLevel(logrus.DebugLevel)
    case "info":
        logrus.SetLevel(logrus.InfoLevel)
    case "warn":
        logrus.SetLevel(logrus.WarnLevel)
    case "error":
        logrus.SetLevel(logrus.ErrorLevel)
    default:
        logrus.SetLevel(logrus.InfoLevel)
    }

    // Set the TextFormatter
    logrus.SetFormatter(&logrus.TextFormatter{
        DisableColors: true,
    })

    logrus.Infoln("xxxxxxx-collector is starting")
}

func main() {
    // Share the configuration
    shared.DatabaseHosts = *databaseHosts
    shared.DatabaseNamespace = *databaseNamespace
    shared.DatabaseUser = *databaseUser
    shared.DatabasePassword = *databasePassword

    // Initialize connection to database
    logrus.Infoln("Initializing connection to database")
    session, err := system.SetupDatabase()
    if err != nil {
        logrus.Fatalln("Error when initializing connection to database : ", err)
    }
    logrus.Infoln("Successfully initialized connection to database")

    // Initialize the DAOs
    logrus.Infoln("Initializing the DAOs")
    indicatorDAO := mongodb.NewIndicatorDAO(session)
    logrus.Infoln("Successfully initialized the DAOs")

    // Initialize the services
    logrus.Infoln("Initializing the services")
    indicatorService := services.NewIndicatorService(indicatorDAO)
    logrus.Infoln("Successfully initialized the services")

    // Setup the providers
    providers, err := system.SetupProviders(*configurationFile)
    if err != nil {
        logrus.Fatalf("Error while setting the providers : %s", err)
    }

    // Initialize a runner
    r := runner.NewRunner(indicatorService, providers, *recycleOffset)
    if err != nil {
        logrus.Fatalln("Error while initializing the runner", err)
    }

    // Initialize CRON
    c := cron.New()
    c.AddFunc("@every 10h", r.Collect)
    c.AddFunc("@daily", r.Recycle)
    c.Start()

    // Handlers
    http.Handle("/metrics", promhttp.Handler())

    // Initialize the goroutine listening to signals passed to the app
    graceful.HandleSignals()

    // Pre-graceful shutdown event
    graceful.PreHook(func() {
        logrus.Infoln("Received a signal, stopping the application")
    })

    // Post-shutdown event
    graceful.PostHook(func() {
        // Stop all the taks
        c.Stop()

        logrus.Infoln("Stopped the application")
    })

    // Listen to the passed address
    logrus.Infoln("Starting the Web server")
    listener, err := net.Listen("tcp", *bindAddress)
    if err != nil {
        logrus.Fatalln("Cannot set up a TCP listener")
    }
    logrus.Infoln("Successfully started the Web server")

    // Run on startup
    if *runOnStartup {
        logrus.Infoln("Running the job on startup")
        r.Recycle()
        go r.Collect()
    }

    // Start the listening
    err = graceful.Serve(listener, http.DefaultServeMux)
    if err != nil {
        logrus.Errorf("Error with the server : %s", err)
    }

    // Wait until open connections close
    graceful.Wait()
}

I would need to put all of this code in this :

var collectorCmd = &cobra.Command{
  Run: func(cmd *cobra.Command, args []string) {
    // Do Stuff Here
    // Do Stuff Here
    // Do Stuff Here
    // Do Stuff Here
  },
}

It is a bit ugly isn't it ?

[jharshman]

If your main function is large, you should look at refactoring it a bit to make it less so. Consider breaking things out into different functions.

[umarcor]

But, the actual main I have is quite huge.. I mean I am declaring the DAOs, the services, the routes...

Yes, you have an ugly non-cobra structure and getting the result you want will require some effort on your side. We can spend much time and discuss a lot about it (in the good sense), but I'm afraid that you already have all the information and examples. You just need to sit down, think about it, read this conversation, think about it again, read the docs, and understand it. Then, spend some hours actually implementing the changes. Honestly, I can do little more, except doing it for you, which is not didactic at all.

I would have to add all of this code into the Run() of the Cobra command ?

No. As I said in my previous comment, you just need to call one function from each Run(), which should require exactly one line.

If you look above, you will see my actual project structure. One main.go for each microservices.

As said, you need to convert each main.go into a different function (name). You will have one function for each microservice, just as you now have one main.go for each of them.

---

Package/lib/file/module with business logic:

import (
  ...
)

var (
  // Move flags to `init()` of the corresponding cobra command and use the syntax supported by cobra
)

func init() {
  // Move content to the `init()` of the corresponding cobra command

  // flag.Parse() NOT required, as it is built in cobra
}

func my_fancy_func_name(args []string) {
  // Do Stuff Here
  // Do Stuff Here
  // Do Stuff Here
  // Do Stuff Here
  // Do Stuff Here
  // Do Stuff Here
  // Do Stuff Here
}

Corresponding cobra command:

var collectorCmd = &cobra.Command{
  Run: func(cmd *cobra.Command, args []string) {
    my_fancy_func_name(args)
  },
}

Standalone entrypoint (equivalent to your current main.go of the microservice):

func main() {
  my_fancy_func_name(os.Args[1:])
}

[fallais]

Thanks for helping, appreciate. I hope this post will help other people.
I thought about what you proposed, I could modify all my main function with this :

func collectorSetup(cmd *cobra.Command, args []string) {
    // code of my main
    // code of my main
    // code of my main
}

Then do this :

var collectorCmd = &cobra.Command{
  Run: collectorSetup,
}

Right ?

[umarcor]

Exactly. That's even cleaner than what I suggested.

EDIT

However, note that collectorSetup will only be usable in a cobra context. If you avoid including cmd *cobra.Command as an argument, then collectorSetup will be usable as a cobra command and/or as a regular package/lib.

[fallais]

Unless I use it like this : collectorSetup(nil, nil).
Am I wrong ?

That being said, I think that choosing Cobra mean, going 100% into it. That would make sens to do what I proposed. But I need to think more.

EDIT : Or I could use a convient function just for Cobra. Like collectorSetupCobra and collectorSetup for simple lib.

I am studying the code of Hugo.

[umarcor]

I think you are on the good path now. All of your latest comments/questions are valid and very pertinent. However, I think that you are the only one that can answer/decide. The specific implementation will depend on your needs and preferences; there is no best solution.

EDIT

After wrapping your head around this, if you feel like proposing a PR to help other users understand these 3-4 different approaches to integrate multiple existing projects into a single cobra project; I'd be happy to review it. Yet, note #959.

[fallais]

I do not often do this kind of work, how could it be done ? I could add, at the root, an examples directory with an example1 inside ? With a basic project based on mine : two microservices called collector and server, a DAO, services, routes and some stuff.

As of today, I fully migrated to Cobra and deployed in production. Quite happy with that. But I am not sure I did the right way, with Logrus for example (logging level setting). Maybe this PR could be a good way to discuss about it ? Actually, I put it in a PersistentPreRunE..

[umarcor]

I do not often do this kind of work, how could it be done ?

Overall, I feel that all the *.md files that are spread across the root (including most of the README) and subdirs doc and cobra could be better located in a specific subdir (probably named docs or site). Moreover, it could be served as a static site (e.g. with hugo). Nonetheless, this is far beyond the scope of your possible contribution.

I could add, at the root, an examples directory with an example1 inside ?

I think that you can name it guide, instead of examples, and I don't think you need another level (example1). This is because I can't imagine any other example right now. Users that want to start a project from scratch do already have enough documentation (https://github.com/spf13/cobra/tree/master/cobra); and your guide will provide guidelines for those who already have multiple projects and want to integrate them with cobra.

With a basic project based on mine : two microservices called collector and server, a DAO, services, routes and some stuff.

I don't think the example/guide needs to be a fully working example (in the sense that it does anything useful). Precisely, I think that putting application specific logic can be misleading for new users. It's ok if you want to pick names as collector, server and so. But I would keep functions as simple as printing some fixed string and/or arguments. The focus should be on explaining the different 2-3 approaches that we commented in the last messages above. Indeed, the contribution can be just a cleanup of this conversation.

But I am not sure I did the right way, with Logrus for example (logging level setting). Maybe this PR could be a good way to discuss about it ? Actually, I put it in a PersistentPreRunE..

I think these are relevant issues, but not to be included in an introduction guide. This is specially so because error handling and log management is not clear enough; hence, we can hardly document something we are not sure about. See #770, #914, #956, #974. You might want to gather the attention of people that participated in those issues in a single place.