Colly: Unmarshalling nested structs

[srph]

Colly is a handy library, thanks by the way! ✨

---

I have this struct which won't work:

package main

import (
    "fmt"
    "encoding/json"
    "log"

    "github.com/gocolly/colly"
)

type Cinema struct {
    Name string `selector:"h2 > a.link > em"`
    Movie *Movie
}

type Movie struct {
    Name        string `selector:"ul > li > div > a > span"`
    URL         string `selector:"ul > li > div > a" attr:"href"`
    Tags        string `selector:".genre"`
    MTRCBRating string `selector:"ul > li > div > div > .mtrcbRating"`
    Duration    string `selector:"ul > li > div > div > .running_time"`
    TimeSlots   []*MovieTimeSlot
}

type MovieTimeSlot struct {
    Time string `selector:".showtimes > span"`
}

func main() {
    cinemas := make([]*Cinema, 0)

    c := colly.NewCollector()

    c.OnHTML("#theatersArea #cinemas .cinema", func(e *colly.HTMLElement) {
        cinema := &Cinema{
            Movie: &Movie{
                TimeSlots: make([]*MovieTimeSlot, 0),
            },
        }
        e.Unmarshal(cinema)
        cinemas = append(cinemas, cinema)
    })

    c.OnRequest(func(r *colly.Request) {
        fmt.Println("Visiting...", r.URL.String())
    })

    c.OnScraped(func(r *colly.Response) {
        data, err := json.Marshal(cinemas)
        if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%s\n", data)
    })

    c.Visit("https://www.clickthecity.com/movies/theaters/lucky-chinatown-mall")
}

Resulting JSON

As you can see, cinema.Movie has blank values.

[
  {
    Name: "Lucky Chinatown Cinema 1",
    Movie: {
      Name: "",
      URL: "",
      Tags: "",
      MTRCBRating: "",
      Duration: "",
      TimeSlots: []
    }
  },
  {
    Name: "Lucky Chinatown Cinema 2",
    Movie: {
      Name: "",
      URL: "",
      Tags: "",
      MTRCBRating: "",
      Duration: "",
      TimeSlots: []
    }
  },
  {
    Name: "Lucky Chinatown Cinema 3",
    Movie: {
      Name: "",
      URL: "",
      Tags: "",
      MTRCBRating: "",
      Duration: "",
      TimeSlots: []
    }
  },
  {
    Name: "Lucky Chinatown Cinema 4",
    Movie: {
      Name: "",
      URL: "",
      Tags: "",
      MTRCBRating: "",
      Duration: "",
      TimeSlots: []
    }
  }
];

---

It happens as long as TimeSlots []*MovieTimeSlot is present. However, if I try to remove movie.TimeSlots or even replace it with TimeSlots []string, it just works:

package main

import (
    "fmt"
    "encoding/json"
    "log"

    "github.com/gocolly/colly"
)

type Cinema struct {
    Name string `selector:"h2 > a.link > em"`
    Movie *Movie
}

type Movie struct {
    Name        string `selector:"ul > li > div > a > span"`
    URL         string `selector:"ul > li > div > a" attr:"href"`
    Tags        string `selector:".genre"`
    MTRCBRating string `selector:"ul > li > div > div > .mtrcbRating"`
    Duration    string `selector:"ul > li > div > div > .running_time"`
    TimeSlots []string `selector:".showtimes > span"`
}

type MovieTimeSlot struct {
    Time string `selector:".showtimes > span"`
}

func main() {
    cinemas := make([]*Cinema, 0)

    c := colly.NewCollector()

    c.OnHTML("#theatersArea #cinemas .cinema", func(e *colly.HTMLElement) {
        // slots := make([]*MovieTimeSlot, 0)
        // movie := &Movie{TimeSlots: slots}
        cinema := &Cinema{
            Movie: &Movie{},
        }
        e.Unmarshal(cinema)
        cinemas = append(cinemas, cinema)
    })

    c.OnRequest(func(r *colly.Request) {
        fmt.Println("Visiting...", r.URL.String())
    })

    c.OnScraped(func(r *colly.Response) {
        data, err := json.Marshal(cinemas)
        if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%s\n", data)
    })

    c.Visit("https://www.clickthecity.com/movies/theaters/lucky-chinatown-mall")
}

Resulting JSON

[
  {
    Name: "Lucky Chinatown Cinema 1",
    Movie: {
      Name: "Mamma Mia! Here We Go Again",
      URL:
        "https://www.clickthecity.com/movies/detail/r8yw0c/mamma-mia-here-we-go-again",
      Tags: "Musical",
      MTRCBRating: "PG",
      Duration: "2 hrs 04 min",
      TimeSlots: ["11:00 AM", "1:40 PM", "4:20 PM", "7:00 PM", "9:30 PM"]
    }
  },
  {
    Name: "Lucky Chinatown Cinema 2",
    Movie: {
      Name: "Mission: Impossible - Fallout",
      URL:
        "https://www.clickthecity.com/movies/detail/lI97Pv/mission-impossible-fallout",
      Tags: "Action, Adventure, Thriller",
      MTRCBRating: "PG",
      Duration: "2 hrs 39 min",
      TimeSlots: ["10:30 AM", "1:30 PM", "4:30 PM", "7:30 PM", "10:30 PM"]
    }
  },
  {
    Name: "Lucky Chinatown Cinema 3",
    Movie: {
      Name: "Mission: Impossible - Fallout",
      URL:
        "https://www.clickthecity.com/movies/detail/lI97Pv/mission-impossible-fallout",
      Tags: "Action, Adventure, Thriller",
      MTRCBRating: "PG",
      Duration: "2 hrs 39 min",
      TimeSlots: ["11:00 AM", "2:00 PM", "5:00 PM", "8:00 PM", "11:00 PM"]
    }
  },
  {
    Name: "Lucky Chinatown Cinema 4",
    Movie: {
      Name: "Skyscraper",
      URL: "https://www.clickthecity.com/movies/detail/4ecp5y/skyscraper",
      Tags: "Action, Crime, Drama",
      MTRCBRating: "PG",
      Duration: "1 hr 53 min",
      TimeSlots: ["12:00 PM", "2:30 PM", "5:00 PM", "7:30 PM", "9:50 PM"]
    }
  }
];

I apologize if I sound ignorant, but does anybody have an idea why? I'm pretty new to Go (and this lib), and I have no concrete idea on how to trace this issue. Thanks!

[asciimoo]

Thanks for the very detailed and clear bug description, I have to check the related Colly code to see what is wrong.

[vosmith]

The current implementation for unmarshalSlice only works for []string as of right now. I'm looking into expanding it to different types.