Colab Pytorch Geometric

Posted by Packy on September 20, 2019

安装pytorch geometric

Windows下配置不成功,编译库失败,转到Ubuntu成功编译。

Windows下配置Pytorch geometric的步骤:

  1. 安装vs2017或者2019,安装配置cuda,cudnn
  2. 还要装一个vc++14 Build tools,以编译下面的库 Install VC++14 from https://go.microsoft.com/fwlink/?LinkId=691126
  3. Install PYG: (Maybe you need to add –user) pip install –verbose –no-cache-dir torch-scatter pip install –verbose –no-cache-dir torch-sparse pip install –verbose –no-cache-dir torch-cluster pip install –verbose –no-cache-dir torch-spline-conv pip install plyfile pip install rdflib pip install torch-geometric

来自 https://github.com/rusty1s/pytorch_geometric/issues/633

比较顺利,按照步骤一步一步就行。

https://zhuanlan.zhihu.com/p/54389036

介绍一下在google Colab 云Python。。下配置PyG的步骤。

千万不要打开两个 Colab 页面,无法同时执行两个笔记本, 执行另外一个,那另一个笔记本内存全部清空!!!

注册Goole账号,在Google Drive 云盘里新建 Colab文件。

1 安装对应版本CUDA

改为GPU模式

笔 记 本 设 置  Python 3  硬 仁 速 器  GPI-J  , O  囗  《 早 存 此 笔 记 本 时 忽 代 码 望 兀 输 土 项

貌似现在Colab自带Cuda了,可以通过!nvcc –version 查看cuda版本。

n..rcc  nvcc:  ——ver on  WIDIA (R) Cuda compiler driver  Copyright (c) 2005—2019 WIDIA Corporation  Built on Wed _ Apr _24_19  Cuda compilation tools, release 10. l, V 10. l. 168

这里选择cuda9.2为例

RAM: 5. 、 1272GB … 3223GB 、 358.27GB  RAM 丨 「 「 「 「 」

12g内存 350g硬盘。。

!wget https://developer.nvidia.com/compute/cuda/9.2/Prod/local_installers/cuda-repo-ubuntu1604-9-2-local_9.2.88-1_amd64 -O cuda-repo-ubuntu1604-9-2-local_9.2.88-1_amd64.deb

!dpkg -i cuda-repo-ubuntu1604-9-2-local_9.2.88-1_amd64.deb

!apt-key add /var/cuda-repo-9-2-local/7fa2af80.pub

!apt-get update

!apt-get install cuda

安装cuda9.2

!wget https://developer.nvidia.com/compute/cuda/9.2/Prod/local_installers/cuda-repo-ubuntu1604-9-2-local_9.2.88-1_amd64 -O cuda-repo-ubuntu1604-9-2-local_9.2.88-1_amd64.deb

!dpkg -i cuda-repo-ubuntu1604-9-2-local_9.2.88-1_amd64.deb

!apt-key add /var/cuda-repo-9-2-local/7fa2af80.pub

!apt-get update

!apt-get install cuda

2 安装Pytorch

pip3 install https://download. pytorch I. vhl  pip3 install https://download. pytorch org/whI/cuIOO/torchdision— 3. vhl  Requirement  Requirement  Requirement  Requirement  Requirement  Requirement  Requirement  Requirement  already  already  already  already  already  already  already  already  satisfied:  satisfied:  satisfied:  satisfied:  satisfied:  satisfied:  satisfied:  satisfied:  torch=l. 1.0 pvtorch. org/whl/culOO/torch—l. l. x86 64. whl in /usr/local/l  nu_mpy in / usr / local/lib/python3. a/dist—packages (from torch=l. 1.0) (l. 16. 4)  torchvision*). pvtorch. org/whl/culOO/torchvision—O. 3. x86 64. whl in  six in /usr/local/lib/python3. a/dist—packages (from torchvision*). 3.0) (l. 12. O)  1.1 in /usr/local/lib/python3. a/dist—packages (from torchvision*). 3.0) (4. 3. O)  1.0 in /usr/local/lib/python3. a/dist—packages (from torchvision*). 3.0) (l. l. O)  nu_mpy in /usr/local/lib/python3. a/dist—packages (from torchvision*). 3.0) (l. 16. 4)  olefile in /usr/local/lib/python3. a/dist—packages (from l. 3. O)  (0. 46)

torch  (torch. __ver sion_)  (torch. cuda.  1. 1.0  True

安装PyG

pip  pip  pip  pip  pip  inst al I  inst al I  inst al I  inst al I  inst al I  — 'verbose  — 'verbose  — 'verbose  — 'verbose  torch—geometric  t or ch—s catt er  —sparse  t or ch—clust er  (optional)

$ pip install –verbose –no-cache-dir torch-scatter $ pip install –verbose –no-cache-dir torch-sparse $ pip install –verbose –no-cache-dir torch-cluster $ pip install –verbose –no-cache-dir torch-spline-conv (optional) $ pip install torch-geometric

!pip install –verbose –no-cache-dir torch-scatter

!pip install –verbose –no-cache-dir torch-sparse

!pip install –verbose –no-cache-dir torch-cluster

!pip install –verbose –no-cache-dir torch-spline-conv

安装torch库的时候,需要用C++编译器编译,对版本要求高,所以windows下很难通过,需要安装对应的编译器版本和CUDA版本。

来自 https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html

Successtully Installed  Cleaning up.  Removed build tracker  Æin/bash: —c: line O:  Æin/bash: —c: line O:  torch—cluster—I  / tmp /pip—r eq—tr ack '  syntax error near Imexpected token  pip install —verbose —no—cache—dir (optional) '  Collecting torch—geometric  Dovmloading pythonhosted. orgZpÄgkgges/cc/b2/19eca95445f4657d6f164e3b57342f138780cca2b4a96a7fla80707d778a/torch geometric—I. 3. O. tax. g Z (102kB)  112kB 9. 5ME/s  Requirement already satisfied: nu_mpy in /usr/local/lib/python3. a/dist—packages (from torch—geometric)  Requirement already satisfied: scipy in /usr/local/lib/python3. a/dist—packages (from torch—geometric)  Requirement already satisfied: networkx in /usr/local/lib/python3. a/dist—packages (from torch—geometric)  Requirement already satisfied: scikit—leaxn in /usr/local/lib/python3. a/dist—packages (from torch—geometric) (O. 21. 2)  Collecting ply-Eile (from torch—geometric)  Dovmloading pythonhosted. orgIp@gkgges 9b 82 d4069cbb49954d44087c37ff616cb423d3e2cOdd276378cdb4af3e3ef2ee ply-file—O. 7. tax. g Z  Requirement already satisfied: pandas in /usr/local/lib/python3. a/dist—packages (from torch—geometric) (O. 24. 2)  Collecting rdflib (from torch—geometric)  Dovmloading pythonhosted. orgZpÄgkgges/3c/fe/630bacb652680f6d481b9febbb3e2c3869194ala5fc3401a4a41195a2f8f /rdflib—4. 2. whl  32.  Requirement already satisfied: h5py in /usr/local/lib/python3. a/dist—packages (from torch—geometric)  Requirement already satisfied: 3.0 in /usr/local/lib/python3. a/dist—packages (from  Requirement already satisfied: Il in /usr/local/lib/python3. a/dist—packages (from (O. 13. 2)  Requirement already satisfied: 5.0 in /usr/local/lib/python3. a/dist—packages (from  Requirement already satisfied: in /usr/local/lib/python3. a/dist—packages (from  (2018.9)  Requirement already satisfied: pypaxsing in /usr/local/lib/python3. a/dist—packages (from  Collecting isodate (from  Dovmloading pythonhosted. orgZpÄgkgges/9b/9f /b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate—O. 6. whl  26. IME/s  Requirement already satisfied: six in /usr/local/lib/python3. a/dist—packages (from  Building wheels for collected packages: torch—geometric, ply-Eile  Building wheel for torch—geometric (setup. py)  Stored in directory: hoot/. cache/pip/wheels/8b/4e/ll/af2fec628d830776b6d49ad4df019d8200d138380dlfae39c2  Building wheel for ply-Eile (setup. py)  Stored in directory: hoot/. cache/pip/wheels/91/3e/ee/e5630efOfd53cedaa6e911ba27e8b40fff034388dlf264bb92  Successfully built torch—geometric ply-Eile  Installing collected packages: ply-file, isodate, rdflib, torch—geometric  Successfully installed isodate—O. 6.0 ply-file—O. 7 rdflib—4. 2. 2 torch—geometric—I. 3.0

无痛安装。。。全程看进度条就可以

下载一下 https://github.com/rusty1s/pytorch_geometric

!git clone https://github.com/rusty1s/pytorch_geometric

git clone Ittps: //github. ccgn/rustyls/pytorch_geometric  Cloning into ' pytorch_geometric'  remote: Enu_merating objects: 15967, done.  remote: Total 15967 (delta O), reused O (delta O), pack—reused 15967  2. 16 MiB 3.94 MiB/s, done.  Receiving objects: 1005 (15967/15967),  Resolving deltas: 1005 (11453/11453), done.

运行一个例子:

[8]  cd pytorch_geometric  / content/pytor ch_geometric  cd exaynples/  / content/pytor  o  : o. 2071,  : o. 4714,  : o. 4714,  : o. 4571,  : o. 5714,  : o. 7429,  : o. 7500,  : o. 7643,  : o. 8071,  : 0.8571,  : 0.8786,  : o. 9000,  : o. 8929,  python g cn  Downloading  Downloading  Downloading  Downloading  Downloading  Downloading  Downloading  Downloading  Processing.  Done  Epoch •  . 001,  Epoch •  . 002,  Epoch •  . 003,  Epoch •  . 004,  Epoch •  . 005,  Epoch •  . 006,  Epoch •  . 007,  Epoch •  . 008,  Epoch •  . 009,  Epoch •  . 010,  Epoch •  . 011,  Epoch •  . 012,  Epoch •  . 013,  i thub.  i thub.  i thub.  i thub.  i thub.  i thub.  i thub.  i thub.  comAimivou_ng/planetoid/raw Imas ter / data/ind.  comAimivou_ng/planetoid/raw Imas ter / data/ind.  comAimivou_ng/planetoid/raw Imas ter / data/ind.  comAimivou_ng/planetoid/raw Imas ter / data/ind.  comAimivou_ng/planetoid/raw Imas ter / data/ind.  comAimivou_ng/planetoid/raw Imas ter / data/ind.  comAimivou_ng/planetoid/raw Imas ter / data/ind.  comAimivou_ng/planetoid/raw Imas ter / data/ind.  cora. x  cora.  allx  cora.  cora.  cora.  allv  cora.  cora.  test. index  cora.  O. 1620,  O. 2120,  O. 2600,  O. 2960,  O. 3500,  O. 5040,  O. 5700,  O. 5700,  O. 5700,  O. 5800,  o. 6140,  Test  Test  Test  Test  Test  Test  Test  Test  Test  Test  Test  Test  Test  : o. 1710  : o. 2270  : o. 2830  : o. 2990  : o. 3790  : o. 5130  : o. 5490  : o. 5490  : o. 5490  : o. 5470  : o. 5890  : o. 6350  : o. 6350

可以运行,说明安装好啦。然后就可以造了。。

!pip3 install https://download.pytorch.org/whl/cu100/torch-1.1.0-cp36-cp36m-linux_x86_64.whl

!pip3 install https://download.pytorch.org/whl/cu100/torchvision-0.3.0-cp36-cp36m-linux_x86_64.whl

!pip install –verbose –no-cache-dir torch-scatter

!pip install –verbose –no-cache-dir torch-sparse

!pip install –verbose –no-cache-dir torch-cluster

!pip install –verbose –no-cache-dir torch-spline-conv (optional)

!pip install torch-geometric

!git clone https://github.com/rusty1s/pytorch_geometric.git

这几行就可以装好Pyg了,每次用都要装一次,估计要花个10来分钟。

!git clone https://github.com/rusty1s/pytorch_geometric

下载文件 数据集等

!wget https://s3-eu-west-1.amazonaws.com/yc-rdata/yoochoose-data.7z

!dir

!7za x yoochoose-data.7z

然后解压, colab的IO性能比较弱,所以解压可能时间比较长

连接谷歌网盘

!apt-get install -y -qq software-properties-common python-software-properties module-init-tools

!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null

!apt-get update -qq 2>&1 > /dev/null

!apt-get -y install -qq google-drive-ocamlfuse fuse

from google.colab import auth

auth.authenticate_user()

from oauth2client.client import GoogleCredentials

creds = GoogleCredentials.get_application_default()

import getpass

!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 grep URL

vcode = getpass.getpass()

!echo {vcode} google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}

# 指定Google Drive云端硬盘的根目录,名为drive

!mkdir -p drive

!google-drive-ocamlfuse drive

复制文件到谷歌网盘

!cp -r /content/processed/yoochoose_click_binary_1M_sess.dataset /content/drive

colab 读入 google drive 的文件

2018年12月16日 11:14:58 code_nie 阅读数 2827

近段时间需要使用google colab训练一些神经网络,但是colab默认是不能读入google drive的数据的,每次都要重新上传,费时费力。所以这篇博客是让colab用户能够使用google drive的工作文件夹

2018年12月亲测有效。

step1

首先需要让colab获得google drive的授权,在google colab里执行如下代码:

!apt-get install -y -qq software-properties-common python-software-properties module-init-tools !add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null !apt-get update -qq 2>&1 > /dev/null !apt-get -y install -qq google-drive-ocamlfuse fuse from google.colab import auth auth.authenticate_user() from oauth2client.client import GoogleCredentials creds = GoogleCredentials.get_application_default() import getpass !google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL vcode = getpass.getpass() !echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}

执行时会打印一个网址和一个授权码的输入框,此时你需要点击网址,登陆google account,授权colab使用你的google drive,然后系统会分配给你一个授权码,你需要将授权码粘贴到输入框里,回车继续。

若打印出success的信息,说明授权成功。

step 2

执行如下代码:

# 指定Google Drive云端硬盘的根目录,名为drive !mkdir -p drive !google-drive-ocamlfuse drive

此时colab中出现drive的文件夹,里面就是你的google drive的根目录文件

step3

然后更换执行的工作文件夹即可。

import os os.chdir(“drive/Colab Notebooks”)

来自 https://blog.csdn.net/code_nie/article/details/85029074