Harbor: Add import image from third docker hub or other private docker registry for Harbor

[frank798]

when we want to push image into harbor,we should pull the image first in cli,and then tag the image,and then push the image(if the project is private,docker login is needed).Does harbor fans think is necessary to add a new feature to harbor to complate these operation by telling harbor which image you want to import,for example,input the image like: index.tenxcloud.com/google_containers/busybox,and then harbor will do these operation automatically.
If you think this feature is needed,I'll do this job.

[hainingzhang]

Thanks for your suggestion. Did you mean to pull from docker hub and then push into Harbor? Is it an external component to harbor or a service in Harbor? We are interested in such topic, please give more details on your thoughts.

[onecool2]

I think frank798's mean, when user want to pull a images which doesn't exist in current Harbor regisrty, Harbor should pull it from other registry or external registry(such as docker hub), then back to this user.
:)
I suppose this feature is very useful, user don't need to use docker hub any more. meanwhile, harbor could be as a repository cache server.

[frank798]

Yes,it is.it is a service in Harbor,it could pull from docker hub or other docker registry.I can provide a UI to operator,user just input the image's address.this feature is organized by project,this means,the image is import into project directly.The import operator process also display for user.

[hainingzhang]

Thanks for your explanation. I agree it is a useful feature. However, to integrate into Harbor, there is still a lot to consider in the whole picture and it could add in more complexity:

1. If the pull/push process is within Harbor, there should be a console to manage this.
2. Need to handle users and privilege of images.
3. If there are replication policies, needs to handle replication.
4. Is this a one-time pull/push or a continuous process? What to do when the source image gets changed?

This is not a small effort. We need more thinking on it.

Alternatively, is it possible to provide an external tool for this purpose?

@frank798

[frank798]

@hainingzhang Thanks for your replay.But I'm afraid my explanation is not clear.Let me try to explain my thoughts clear.
Firstly,Let's take a look at what dose user do when they want to pull a image from docker hub or other private registry,there are four steps to do,all there steps would do by docker CLI
　　step 1
　　 　 docker pull busybox
　　step 2:
　　 　docker tag <IMAGE_ID> your_domain_name.com/library/busybox
　　 step 3:
　　 　docker login your_domain_name.com
　　step 4:
　　 　 docker push your_domain_name.com/library/busybox
　
　　After these step,we would find the image in project "library"

　　So, my thoughts is let harbor do these steps for user,user just input the image name.This feature is a one-time process and seems like user do by docker cli. Let me further explain this feature.
　　
　　　This feature could be split into six steps,and provided by API for UI.
　　step 1:
　　　　User select a project, the current user must have right to push.
　　step 2:
　　　　User input image name,and then click button to start import image
　　step 3：
　　　　The process pull image from docker hub
　　step 4:
　　　　After pull image complete,the process tag image.the tag's format is: 　　　your_domain_name.com/project_name/image_name:tag
　　step 5:
　　　　 push image to registry
　　step 6:
　　　　After push image complete.the process would delete image from local.

　　The whole process complete,user can find the image in repositories.

[hainingzhang]

@frank798 Thanks for the clarification. I agree it is a useful feature, it could be considered as "import images from other sources".

The complexity is in Step 3, it may take a quite some time to finish pulling depending on the network and image size. Also it may fail due to various errors. It is an asynchronous process, so Harbor needs to handle all these situations, which is very similar to Harbor's replication process. If the user cannot monitor the progress during the pulling, it will be very user unfriendly. For this reason, Harbor should handle(manage) the pulling process gracefully.

It seems your suggestion is to run a script at the backend of Harbor, but it is a bit hard to manage that.
What do you think?

[frank798]

@hainingzhang Yes,pull image would take a quite some time.Do you mean it's better to handle this by SM like Harbor's replication process?

Actually,the Docker engine provide remote api to help me to complete this feature. I've test the api, the response is similar to pull image by docker CLI. So,I think I can use these api to do this feature.

[Vidanez]

Hello, I'm also interested on this feature. Do you have any updates?

[luisyonaldo]

Any progress on this?

I have a similar need. I want to restrict access to users inside the kubernetes cluster to be able to pull images from public dockerhub (due to security concerns).

Desired behavior is as follow: in case they want a public image, that image needs to be approved and then pulled from dockerhub and pushed to harbor. So basically what we want is a way to specify "approved" public images so harbor syncs them with dockerhub (download then from dockerhub).

Any idea is this is currently possible or planned? Or a good alternative/solution to implement it? @hainingzhang

Thanks!

[jakirpatel]

Hello,

We have a current requirement for this kind of feature. Some of the images which are available on docker hub need to be replicated inside the library project of the harbor.

When a user tries to download some public image lets stay XYZ:latest with the harbor that means.

docker pull yourdomain.com/library/XYZ:latest

If the image is not available with Harbor then harbor should try to find and fetch it from docker hub. Also, it should do the job of handling the project of the image. For my case, I 'll prefer library repo which is public.

Thanks !

[ghost]

Hi @jakirpatel and @luisyonaldo – still on our todo list. Contributions are welcomed and we're happy to offer help to anyone interested in implementing this.

[luisyonaldo]

@clouderati I'll check current mirroring logic in harbor this weekend. I'll probably need some help but I'll start with trying to understand current logic and later on try to add dockerhub mirror functionality.

[ghost]

Awesome! Thanks @luisyonaldo. Please do feel free to ping me and the team either here or on Slack (#harbor-dev on CNCF Slack); happy to help you through the process.

If possible, let's try to keep things generic enough so that we can mirror from other Docker registry nodes, too.

/cc @reasonerjt @bradmeiseles @pauldul

[pandom]

Major issue here for a few customers who are using or evaluating harbor. Happy to speak publicly or internally (VMware) about these customers.

[ArjonBu]

Any news regarding this feature?

[wy65701436]

It’s under development, hopefully we can enable it in the next release.

[8ear]

Is there any existing pull request or branch? For help.
Hi @luisyonaldo have you any problems to setup it?

[ywk253100]

Replicating images from Docker Hub or Docker registry to Harbor is available in 1.8. Closing this issue.

[mantshar1990]

I have requirement i want to pull images from docker CLI using harbor registry, how to restrict the images can be pulled by the specific user, images access should not allowed to other user ?

example --

myregistry/user1/abc:latest
myregistry/user2/xyz:latest

if user1 perform pull for myregistry/user1/abc:latest, user1 can able to pull the images.
if user1 perform pull for myregistry/user2/xyz:latest, user1 should not allow to pull the images.

same condition applies for user2 also.

can i get the solution for this using harbor registry ?.