其他 on Eryk's Blog

gitlab 持续集成

Mon, 05 Feb 2018 11:35:00 +0000

安装gitlab-runner

https://docs.gitlab.com/runner/install/index.html

国内镜像地址

https://mirrors.tuna.tsinghua.edu.cn/help/gitlab-ci-multi-runner/

注册Runner

https://docs.gitlab.com/runner/register/

步骤

注意：用户为root用户

root@sw:/home/sw# gitlab-runner register xuqi
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://10.168.2.114/

Please enter the gitlab-ci token for this runner:

输入gitlab的token

Please enter the gitlab-ci description for this runner:
[sw]: xuqi
Please enter the gitlab-ci tags for this runner (comma separated):

Whether to lock the Runner to current project [true/false]:
[true]: true
Registering runner... succeeded                     runner=wzb81TcG
Please enter the executor: docker, parallels, shell, virtualbox, docker+machine, kubernetes, docker-ssh, ssh, docker-ssh+machine:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

注意： token位置在http://localhost/xxx/xxx/settings/ci_cd 页面里的Runners settings里

jupyter性能分析工具

Tue, 05 Dec 2017 12:18:00 +0000

magic functions:

%time: 记录script运行时间
%timeit: 循环n次，记录平均执行时间
%run & %prun: 分析script的执行效率
%lprun: 分析一条语句在function中执行效率
%mprun: 查看script脚本使用内存量
%memit: 循环n次，记录平均内存使用量

安装

pip install jupyter_contrib_nbextensions
pip install line-profiler
pip install psutil
pip install memory_profiler

jupyter load 插件

1
2

%load_ext line_profiler
%load_ext memory_profiler

Time Profiling

time & timeit

In [7]: %time {1 for i in xrange(10*1000000)}
CPU times: user 0.72 s, sys: 0.16 s, total: 0.88 s
Wall time: 0.75 s

In [8]: %timeit 10*1000000
10000000 loops, best of 3: 38.2 ns per loop

In [9]: %timeit -n 1000 10*1000000
1000 loops, best of 3: 67 ns per loop

%prun

In [10]: from time import sleep

In [11]: def foo(): sleep(1)

In [12]: def bar(): sleep(2)

In [13]: def baz(): foo(), bar()

In [14]: %prun baz()
7 function calls in 3.001 seconds

Ordered by: internal time

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    3.001    1.500    3.001    1.500 {time.sleep}
     1    0.000    0.000    3.001    3.001 <ipython-input-17-c32ce4852c7d>:1(baz)
     1    0.000    0.000    2.000    2.000 <ipython-input-11-2689ca7390dc>:1(bar)
     1    0.000    0.000    1.001    1.001 <ipython-input-10-e11af1cc2c91>:1(foo)
     1    0.000    0.000    3.001    3.001 <string>:1(<module>)
     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Memory Profiling

%mprun

In [17]: %mprun -f foo foo(100000)
Filename: foo.py

Line #    Mem usage    Increment   Line Contents
================================================
     1    20.590 MB     0.000 MB   def foo(n):
     2    20.590 MB     0.000 MB       phrase = 'repeat me'
     3    21.445 MB     0.855 MB       pmul = phrase * n
     4    25.020 MB     3.574 MB       pjoi = ''.join([phrase for x in xrange(n)])
     5    25.020 MB     0.000 MB       pinc = ''
     6    43.594 MB    18.574 MB       for x in xrange(n):
     7    43.594 MB     0.000 MB           pinc += phrase
     8    41.102 MB    -2.492 MB       del pmul, pjoi, pinc

在jupyter中运行报如下错误：

使用树莓派+kodi搭建多媒体环境

Sun, 03 Dec 2017 14:26:00 +0000

制作启动盘

下载 RASPBIAN STRETCH WITH DESKTOP，地址如下

https://www.raspberrypi.org/downloads/raspbian/

使用SDFormatter 格式化sd卡
使用win32diskimager将 .img文件写入sd卡

安装kodi

参考如下步骤

数据预处理-清洗转换

Thu, 30 Nov 2017 15:39:00 +0000

转自：http://www.dataivy.cn/blog/data_etl/?wt_tb=1%7C1511709850969

当数据质量校验完成后，针对有问题的数据要进行的是数据清洗和转换，另外还包括对正常数据的转换。数据清洗的主要作用包括：纠正错误、删除重复项、统一规格、修正逻辑、转换构造和数据压缩。

linux用户进程执行过程被系统kill掉了

Fri, 22 Sep 2017 15:16:00 +0000

系统日志显示

Sep 22 12:40:44 gpu01 systemd[1]: Stopping User Manager for UID 1001...
Sep 22 12:40:44 gpu01 systemd[1453]: Stopped target Default.
Sep 22 12:40:44 gpu01 systemd[1453]: Stopped target Basic System.
Sep 22 12:40:44 gpu01 systemd[1453]: Stopped target Timers.
Sep 22 12:40:44 gpu01 systemd[1453]: Stopped target Sockets.
Sep 22 12:40:44 gpu01 systemd[1453]: Stopped target Paths.
Sep 22 12:40:44 gpu01 systemd[1453]: Reached target Shutdown.
Sep 22 12:40:44 gpu01 systemd[1453]: Starting Exit the Session...
Sep 22 12:40:44 gpu01 systemd[1453]: Received SIGRTMIN+24 from PID 28725 (kill).
Sep 22 12:40:44 gpu01 systemd[1]: Stopped User Manager for UID 1001.
Sep 22 12:40:44 gpu01 systemd[1]: Removed slice User Slice of xxx.

google到如下解释： https://serverfault.com/questions/774491/what-is-sigrtmin24-in-syslog

10分钟搭建github博客

Fri, 18 Aug 2017 10:16:44 +0000

搭建github博客，参考如下步骤

https://pages.github.com/

注册域名，修改域名DNS地址

找个域名服务商注册个域名，我使用的是godaddy，如果以后需要指向国内空间，建议在国内域名服务商注册。

注册好域名后，可以直接将域名cname到github博客地址，也可以使用dnspod.com的dns解析域名，在域名管理中把dns地址修改dnspod的dns：