Pytorch_geometric: Unpooling layers and generative architectures

[robclouth]

Hey there! Thanks for your hard work with this project, it's super cool. In the end of the splineCNN paper, it was mentioned that unpooling layers were a future possibility. I'd love to try generating graphs from a continuous latent space. Any ideas for where I might start? Any projects like this one that you're aware of with generative capabilities? Thanks.

[robclouth]

Oh wait I just saw the other issue on this topic. Sorry for cluttering the issues.

[robclouth]

I've done some research and this paper and code could be interesting to you as it uses decoding.

https://github.com/Ruiqi-Hu/ARGA

[rusty1s]

Hi, the paper you mentioned seems to be relatively similar to VGAEs, where the variational autoencoder is replaced with an adversarial autoencoder. This can be built in PyTorch Geometric within a few lines of code. I might make an example to showcase this. Other papers I am aware of are GraphGANs or NetGANs, but I am sure there are tons of other ones.

[jlevy44]

@rusty1s @robclouth I second this. I'd really like an exploration of generative methods using torch_geometric.

[jlevy44]

I can help write the code if you are busy. If you have practical recommendations for writing it, I'd love to hear before this becomes a time sink.

[jlevy44]

I may just work on implementing ARGA if you haven't already.

[jlevy44]

I have a small network that I'd like to compare to ERGM and LatentNet models

[rusty1s]

Since request is growing, it's definitively time to integrate graph generation methods into PyG. VGAE + ARGA models/examples seem to be a good starting point before implementing more time-consuming ones. Maybe we should add a new subpackage torch_geometric.nn.models to integrate those?

@jlevy44 Would love to see you making the first move. If you have any problems, feel free to reach out. Otherwise, I could implement it in the next week.

[jlevy44]

Sure. I'll give ARGA a shot and let you know if I run into issues. May take me more than a week though since I'm leaving for a trip tomorrow.

[jlevy44]

Do you have any recommendations for similar architectures that operate autoregressively? Maybe I can just define a loss function that takes into account random walks. Goal is to capture transitive properties of graph.

[jlevy44]

Maybe some combination between netGAN and ARGA would be nice to implement in the future.

[rusty1s]

GraphRNN?

[jlevy44]

Yep. I agree with this. I've read the paper, been wanting to implement, how easy is it to implement?

[jlevy44]

I haven't tested my code yet, but does this seem like it's on the right track??:

from torch.autograd import Variable
from copy import deepcopy

def calc_loss(A_orig, A_new, d_real, d_fake, weights):# weights = (float(A_orig.shape[0]**2-A_orig.sum())/A_orig.sum()).flatten()
    dc_real_loss=nn.BCELoss(reduction='mean')(d_real,torch.ones(d_real.size()))
    dc_fake_loss=nn.BCELoss(reduction='mean')(d_fake,torch.zeros(d_fake.size()))
    #print(sum(A_new),sum(A_orig))
    A_loss = nn.BCELoss(reduction='sum',weight=torch.tensor(weights,dtype=torch.float))(A_new,A_orig)
    #print(dc_real_loss.item(),dc_fake_loss.item(),A_loss.item())
    total_loss = dc_real_loss + dc_fake_loss + A_loss
    return total_loss

def calc_weights(A):
    return (float(A.shape[0]**2-A.sum())/A.sum()).flatten()

class EncoderGCN(nn.Module):
    def __init__(self, n_total_features, n_latent, p_drop=0.):
        super(EncoderGCN, self).__init__()
        self.p_drop = p_drop
        self.n_total_features = n_total_features
        self.conv1 = nn_geom.GCNConv(self.n_total_features, 11)
        self.f1=nn.Sequential(nn.ReLU(),nn.Dropout(self.p_drop))
        self.conv2 = nn_geom.GCNConv(11, 11)
        self.f2=nn.Sequential(nn.ReLU(),nn.Dropout(self.p_drop))
        self.conv3 = nn_geom.GCNConv(11,n_latent)
        torch.nn.init.orthogonal_(self.conv1.weight)
        torch.nn.init.orthogonal_(self.conv2.weight)
        torch.nn.init.orthogonal_(self.conv3.weight)

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x = self.f1(self.conv1(x, edge_index)) # X,A
        x = self.f2(self.conv2(x, edge_index))
        x = self.conv3(x, edge_index)
        return x

class DecoderGCN(nn.Module):
    def __init__(self):
        super(DecoderGCN, self).__init__()
        self.output_layer = nn.Sigmoid()

    def forward(self, z):
        A=torch.mm(z,torch.t(z))
        #print(A)
        A=self.output_layer(A)
        return A

class Discriminator(nn.Module):
    def __init__(self, n_input):
        super(Discriminator, self).__init__()
        self.fc1 = nn.Sequential(nn.Linear(n_input,40), nn.ReLU())
        self.fc2 = nn.Sequential(nn.Linear(40,30), nn.ReLU())
        self.fc3 = nn.Sequential(nn.Linear(30,1),nn.Sigmoid())

    def forward(self, z):
        z=self.fc1(z)
        z=self.fc2(z)
        out=self.fc3(z)
        return out

class ARGA(nn.Module):
    def __init__(self, n_total_features, n_latent):
        super(ARGA, self).__init__()
        self.encoder = EncoderGCN(n_total_features, n_latent)
        self.decoder = DecoderGCN()
        self.discriminator = Discriminator(n_latent)

    def forward(self, x):
        z_fake = self.encoder(x)
        z_real=torch.randn(z_fake.size())
        #print(z_fake)
        A = self.decoder(z_fake)
        d_real=self.discriminator(z_real)
        d_fake=self.discriminator(z_fake)
        return A, d_real, d_fake

    def simulate(self, x):
        z_fake = self.encoder(x)
        A = self.decoder(z_fake)
        return A

class ARGA_Model:
    def __init__(self, optimizer, n_epochs, model=ARGA(20,30), n_nodes=20, n_latent=30, weights=None, scheduler_opts=dict(scheduler='warm_restarts',lr_scheduler_decay=0.5,T_max=10,eta_min=5e-8,T_mult=2)):
        self.loss_fn = lambda A_orig, A_new, d_real, d_fake: calc_loss(A_orig, A_new, d_real, d_fake, weights)
        self.model=model#(n_nodes,n_latent)
        self.optimizer = optimizer
        self.scheduler = Scheduler(optimizer, scheduler_opts)
        self.n_epochs = n_epochs
        self.flatten = lambda x: torch.flatten(x)
        self.sigmoid_fn = nn.Sigmoid()

    def train(self, data): # one
        self.model.train()
        self.total_loss = []
        self.total_lr = []
        A_real = return_adjacency(data)
        A_real = self.flatten(A_real).float()
        self.best_model=None
        for epoch in range(self.n_epochs):
            A_fake, d_real, d_fake = self.model(data)
            A_fake = self.flatten(A_fake)
            loss = self.loss_fn(A_real, A_fake, d_real, d_fake)
            loss.backward()
            self.optimizer.step()
            self.total_lr.append(self.scheduler.get_lr())
            self.scheduler.step()
            loss_float=loss.item()
            print(loss_float)
            self.total_loss.append(loss_float)
            if loss_float <= min(self.total_loss):
                self.best_model = deepcopy(self.model)
        self.model = self.best_model


    def simulate(self, data):
        self.model.eval()
        data=self.model.simulate(data)
        print(data)
        return self.sigmoid_fn(data).detach().numpy()#.item() # return_adjacency(

import torch.optim
n=covariates.shape[1]
n_latent=2
model=ARGA(n,n_latent)
trainer=ARGA_Model(optimizer=torch.optim.Adam(params=model.parameters(),lr=1e-3,weight_decay=5e-1), n_epochs=100, model=model, n_nodes=n, n_latent=n_latent, weights=calc_weights(A))
trainer.train(graph.graph)
simulated_graph=trainer.simulate(graph.graph)

class SingleGraph(Dataset):
    def __init__(self, graph_dict, outcome_col=None): # if None do all vs all
        """graph_dict=dict(A=nx.Graph(...),X=pd.DataFrame(..., index=[node labels], columns=[covariates]))"""
        self.graph = graph_dict
        self.all_labels = np.array(list(self.graph['A'].nodes()))
        self.graph=graph_to_data(self.graph, outcome_col)
        self.length = len(self.all_labels)
        self.y = outcome_col
        if self.y==None:
            self.y = np.ones((self.length,))

    def __getitem__(self, i):
        return self.graph

    def __len__(self):
        return self.length

[jlevy44]

@rusty1s If you could just check superficially from the paper: https://arxiv.org/pdf/1802.04407.pdf

[jlevy44]

Not sure if I should employ a data loader if it is just for a single graph...

[jlevy44]

I'll try to wrap this up after you provide some feedback. If anything, we can make a PR and continue this discussion over there.

[jlevy44]

I'll try to test in the meanwhile and see if I can get anything to work.

[rusty1s]

You are fast :) The basic idea looks correct to me. However, it would be really cool to also provide mini-batch support (this should be possible with torch_geometric.utils.to_dense_batch() after encoding). I can give you more in-depth feedback tomorrow.

Edit: You do not need a dataloader for a single graph (similar to how all Cora examples in the examples/ directory simply operate on dataset[0]).

[jlevy44]

Okay thanks. I just put this together in about an hour and a half so I don't expect any of this to be functional. Thanks! If you could give me a list of things to do to wrap it up, that'd be awesome.

Hope we can PR it soon so I can start using it in my work.

[jlevy44]

Ah I see I made a few mistakes up at the sequential layers of the encoder (can't accept edge_list because I wrapped it into sequential). I'm sure you'll pick it all up. I'm currently debugging, working my way through this code. Looking forward to your feedback!

[jlevy44]

Actually got it to run the simulate and forward pass method, so this could be promising.

[jlevy44]

I was able to train and test the model. Results didn't look so great though. Train loss dropped from 50k to 1k, but simulated graph appeared to be quite off (all of the values dropped to near 0.5). I think it's an implementation error, but would appreciate the direction.

[jlevy44]

I wonder if the adversarial regularization needs to be given higher weight, similar to VAE KL loss for example. Once this works, I'll add KL loss and shoot for VGAE.

[rusty1s]

Cool! Just wondering why you use reduction='sum' in the reconstruction loss. This is not done in the official implementation. Can you open a pull request so that we can discuss the details there (and I can check out the current version)?

[jlevy44]

Sure, sounds good.

[jlevy44]

Shall we close this thread?