https://www.bigdick.live • This is a web feed, also known as an Atom feed. Subscribe by copying the URL from the address bar into your newsreader. Visit About Feeds to learn more and get started. It's free. Visit website Recent posts Updated on $DATE date Yu Sun Yu Sun's personal site for bioinformatics, data science, software engineering, public resume, projects, and posts. Zola2017-06-19T00:00:00+00:00https://www.bigdick.live/en/atom.xml 2017-06-19T00:00:00+00:00 2017-06-19T00:00:00+00:00 Yu Sun https://www.bigdick.live/en/blog/maintaining-aur-packages/ <h2 id="generating-an-ssh-key-pair">Generating an SSH key pair</h2> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">ssh-keygen</span><span class="z-string z-punctuation z-definition z-string"> -t ecdsa -C &quot;</span><span class="z-punctuation">$(</span><span class="z-entity z-name z-function">whoami</span><span class="z-punctuation">)</span><span class="z-string">@</span><span class="z-punctuation">$(</span><span class="z-entity z-name z-function">hostname</span><span class="z-punctuation">)</span><span class="z-string">-</span><span class="z-punctuation">$(</span><span class="z-entity z-name z-function">date</span><span class="z-string"> -I</span><span class="z-punctuation">)</span><span class="z-punctuation z-definition z-string">&quot;</span></span></code></pre> <p>During the process, it’s highly recommended to specify a file name for this ssh key, for you may have more keys (for <em>GitHub</em>, etc) in the future. Also you should remember the passphrase.</p> <h2 id="authentication">Authentication</h2> <p>Suppose my key pair is named as <code>aur_ecdsa</code> and <code>aur_ecdsa.pub</code>. Go to the <a class="external" rel="noopener external" target="_blank" href="https://aur.archlinux.org">index page</a> of <em>AUR</em> and choose “My account”, then enter the content of <code>aur_ecdsa.pub</code>.</p> <p>Edit the configuration file: <code>vim ~/.ssh/config</code>, then add lines below.</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>Host aur.archlinux.org</span></span> <span class="giallo-l"><span> IdentityFile ~/.ssh/aur_ecdsa</span></span> <span class="giallo-l"><span> User aur</span></span></code></pre> <blockquote class="markdown-alert-note"> <p>The value of the <code>User</code> should always be <code>aur</code> but not others.</p> </blockquote> <p>Save it!</p> <h2 id="modify-the-package">Modify the package</h2> <p>In order to maintaining the <em>Git</em> repository for an existing package, simply <code>git clone</code> the remote repository with the corresponding name:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">git</span><span class="z-string"> clone git+ssh://aur@aur.archlinux.org/package_name.git</span></span></code></pre> <p>Use <code>vim PKGBUILD</code> to modify the content of <code>PKGBUILD</code> file. Then try<code>makepkg --printsrcinfo &gt; .SRCINFO</code> to create metadata file of the source. Finally, <code>stage -&gt; commit -&gt; push</code>:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">git</span><span class="z-string"> add PKGBUILD .SRCINFO</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">git</span><span class="z-string z-punctuation z-definition z-string"> commit -m &quot;useful commit message&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">git</span><span class="z-string"> push origin master</span></span></code></pre> 2017-04-20T00:00:00+00:00 2017-04-20T00:00:00+00:00 Yu Sun https://www.bigdick.live/en/blog/working-on-linux/ <h2 id="os-installation">OS Installation</h2> <ol> <li> <p>Preparation of a USB media.</p> <ol> <li>Download the proper <a class="external" rel="noopener external" target="_blank" href="https://manjaro.org/products/download/x86/"><em>Manjaro KDE Edition</em></a> disc image (ISO file).</li> <li>Check your device list using <code>sudo fdisk -l</code>.</li> <li>Run command as <code>sudo dd if=~/Downloads/manjaro-kde-17.0.1-stable-x86_64.iso of=/dev/sdc</code> (You may replace the ISO file name and target device name by yourself). This will take even over 10 minutes. For Windows user, it is highly recommended to use <em>Rufus</em>, which can automatically process with more than one partition. Set the options as:</li> </ol> <p><img src="/legacy-assets/my-website/img/post_img/rufus_main.png" alt="rufus settings" /></p> <p>When it comes with a dialog box for choosing writing mode, choose as:</p> <p><img src="/legacy-assets/my-website/img/post_img/rufus_dd.png" alt="rufus dd" /></p> </li> <li> <p>Restart your PC and press <code>F12</code> to choose boot from <em>USB flash driver</em>.</p> </li> <li> <p>Start Installation. Choose correct partition table type(<code>GPT</code> for UEFI supported motherboard, with manually choose UEFI boot later. <code>MBR</code> for others). You never need set a mount point of <code>/home</code> at this step, for it should be mounted with another hard disk who will storage data. Volume of these partitions should be set as below:</p> <table><thead><tr><th>Directories</th><th>Volume</th><th>File system</th></tr></thead><tbody> <tr><td>/boot</td><td>512M <sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">1</a></sup></td><td>ext4</td></tr> <tr><td>swap</td><td>2048M <sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">2</a></sup></td><td>Linux-swap</td></tr> <tr><td>/</td><td>remaining space <sup class="footnote-reference" id="fr-3-1"><a href="#fn-3">3</a></sup></td><td>ext4</td></tr> </tbody></table> </li> <li> <p>After installation, reboot and try mount <code>/home</code>.</p> <ol> <li>Use <code>sudo fdisk -l</code> to check all mountable devices.</li> <li>Make partition: <strong>Suppose another hard disk is recognized as <code>/dev/sdb</code></strong> (for example). If its volume less than 2TB, you can simply use <code>fdisk</code> to make partition table for it. Else, you may use another advanced tool like <code>parted</code>. Run <code>parted /dev/sdb</code> to start, you will get a <code>(parted)</code> prompt. Here we want to create only one partition, whose type is <code>primary</code> with <code>ext4</code> file system, just type <code>mkpart primary ext4 0% 100%</code>.</li> </ol> </li> </ol> <blockquote class="markdown-alert-note"> <p>Some users may notice that <code>print</code> can be used to show information of current device, then they may use <code>mkpart primary ext4 0 8TB</code> (for example) to build partition table, which sometimes raise a warning message:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>Warning: The resulting partition is not properly aligned for best performance.</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>Ignore/Cancel?</span></span></code></pre> <p>The warning means the partition start is not aligned. Anyway, <code>0KB</code> is not real start of your disk.</p> <p>What you should do is to remove all partitions on the disk then run the command like <code>./parted_mkpart_calc.sh sdb</code> (the shell script can be obtained <a href="/legacy-assets/my-website/code/parted_mkpart_calc.sh">here</a>), then you will see imformations like:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>Using default 1 MiB default alignment in calc</span></span> <span class="giallo-l"><span>Calculated alignment for /dev/sdb (gpt) is: 2048s</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>If you would be root, you could create partition with:</span></span> <span class="giallo-l"><span># parted /dev/sdb mkpart [name] [type] 2048s 15624996863s</span></span> <span class="giallo-l"><span>Verify partition alignment with:</span></span> <span class="giallo-l"><span># parted /dev/sdb align-check optimal 1</span></span> <span class="giallo-l"><span>Should return: 1 aligned</span></span></code></pre> <p>So you can try <code>parted /dev/sdb mkpart primary ext4 2048s 15624996863s</code> and finally run <code>parted /dev/sdb align-check optimal 1</code> for checking.</p> </blockquote> <ol start="5"> <li> <p>Run <code>mkfs.ext4 -F /dev/sdb1</code> to reformat partition (<code>parted</code>’s <code>ext4</code> parameter doesn’t works, I don’t know why yet).</p> </li> <li> <p><code>mount /dev/sdb1 /home</code> to mount new disk as <code>/home</code>. To make this mount point auto-mounted, add <code>/dev/sdb1 /home ext4 defaults,noatime 0 2</code> at the last line of <code>/etc/fstab</code>, then restart.</p> </li> <li> <p>To configure auto-starting services with <code>systemd</code> (use the <code>sshd.service</code> as example, you may also add <code>teamviewerd</code> and <code>rstudio-server.service</code> for auto-start them):</p> <ol> <li> <p>Use the command like <code>sudo systemctl enable sshd.service</code> to enable the service, and this should create a symlink in <code>/etc/systemd/system/multi-user.target.wants/</code> that looks like the following (do <strong>NOT</strong> create this manually):</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>lrwxrwxrwx 1 root root 38 Aug 1 04:43 /etc/systemd/system/multi-user.target.wants/service.service -&gt; /usr/lib/systemd/system/service.service</span></span></code></pre></li> <li> <p><code>vim /etc/systemd/system/multi-user.target.wants/sshd.service</code> to check the content of the service file. Make sure the file contains a line like <code>Restart=always</code> under the <code>[Service]</code> section of the file to enable the service to respawn after a crash.</p> </li> <li> <p>Finally, reload the <code>systemd daemon</code>, followed by a restart of the service:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> systemctl daemon-reload</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> systemctl restart sshd.service</span></span></code></pre></li> </ol> </li> <li> <p>If you have /home directory on KDE platform before, you may run <code>cd; rm -rf .cache .config .kde4 .local</code> to reset your KDE environment.</p> </li> </ol> <h2 id="autostart">Autostart</h2> <p>To make programs (Yakuake, etc.) autostart: Alt + F2 -&gt; autostart -&gt; Add Program…</p> <h2 id="r-e-p-o">R!E!P!O!</h2> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"># Testing and optimizing mirrors</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pacman-mirrors -g</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pacman-optimize</span><span class="z-punctuation"> &amp;&amp;</span><span class="z-entity z-name z-function"> sync</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"># Updating OS</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pacman -Syyu</span></span></code></pre> <blockquote class="markdown-alert-note"> <p>For some Chinese users, generating mirror list may raise an error (probably caused by network), you can use <code>sudo pacman-mirrors -g -c china</code> instead.</p> </blockquote> <p>Then, for all Chinese users, it is better to add archlinuxcn mirror for <em>pacman</em>. Enter <code>sudo nano /etc/pacman.conf</code> then:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>[archlinuxcn]</span></span> <span class="giallo-l"><span>SigLevel = Optional TrustedOnly</span></span> <span class="giallo-l"><span>Server = https://mirrors.ustc.edu.cn/archlinuxcn/$arch</span></span></code></pre> <p>You should put it <strong>bebind</strong> the main repos (for example, core) to prevent conflicts. Then:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pacman -Syyu</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pacman -S archlinuxcn-keyring</span></span></code></pre> <p>Finally, use <em>aurman</em> for enhanced package manager:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pacman -S aurman</span></span></code></pre> <blockquote class="markdown-alert-warning"> <p><strong>DO NOT</strong> use <em>yaourt</em> any more nowadays. It’s <a class="external" rel="noopener external" target="_blank" href="https://wiki.archlinux.org/title/AUR_helpers">out of date</a>.</p> </blockquote> <p>To upgrage the whole system, you should run <code>aurman -Syu</code> weekly. If there’s message like this:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>~~ the following packages are neither in known repos nor in the aur</span></span> <span class="giallo-l"><span>:: pcmciautils-018-8</span></span></code></pre> <p>Then you can run <code>pacman -Rns $(pacman -Qtdq)</code> to remove all unused packages.</p> <blockquote class="markdown-alert-note"> <p>For some Chinese users, <code>aurman</code> may raise connection error, you can edit <code>~/.config/aurman/aurman_config</code> to set longer timeout for it.</p> </blockquote> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>[miscellaneous]</span></span> <span class="giallo-l"><span>aur_timeout=100</span></span></code></pre><h2 id="change-default-dns-server">Change default DNS server</h2> <p>Install dnsutils first, then check DNS address.</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pacman -S dnsutils</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">dig</span><span class="z-string"> www.baidu.com</span></span></code></pre> <p>Edit <code>/etc/resolv.conf</code> to change DNS server by adding these lines:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span># Google nameservers</span></span> <span class="giallo-l"><span>nameserver 8.8.8.8</span></span> <span class="giallo-l"><span>nameserver 8.8.4.4</span></span></code></pre> <p>Use <code>sudo chattr +i /etc/resolv.conf</code> to set write-protection for this file (<strong>generally it will be reset during reboot</strong>).</p> <h2 id="set-users-as-sudoers">Set users as sudoers</h2> <p>By typing <code>sudo visudo</code>, you can change sudoers in <em>vi</em> mode, then use <code>/root</code> with <code>n</code> (perhaps more than one time) to change the position of cursor to the <code>User privilege specification</code> section. Press <code>yyp</code> to copy &amp; paste the current line, then <code>dft</code> to delete username ‘root’, finally add the new sudoer’s username. Press <code>:wq</code> to save and exit. Now this section of configuration file seems like:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>##</span></span> <span class="giallo-l"><span>## User privilege specification</span></span> <span class="giallo-l"><span>##</span></span> <span class="giallo-l"><span>root ALL=(ALL) ALL</span></span> <span class="giallo-l"><span>ysun ALL=(ALL) ALL</span></span></code></pre> <blockquote class="markdown-alert-note"> <p>If you want add a sudoer who never need password to run commands as root, you can set as:</p> </blockquote> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>ysun ALL=(ALL) NOPASSWD:ALL</span></span></code></pre> <p>You should only use <strong>one space</strong> as separator in this section (perhaps in the whole file) for correctly recognized.</p> <h2 id="install-and-configure-softwares">Install and Configure softwares</h2> <h3 id="vim">VIM</h3> <p><code>sudo pacman -S vim</code></p> <h3 id="teamviewer">TeamViewer</h3> <p><code>sudo pacman -S teamviewer</code></p> <blockquote class="markdown-alert-note"> <p>To make TeamViewer work properly under <em>Fedora 25</em> with <em>Gnome 3</em>:</p> <ol> <li>Open the file <code>/etc/gdm/custom.conf</code>.</li> <li>Uncomment or add the line <code>WaylandEnable=false</code>.</li> <li>Save and reboot the system.</li> </ol> </blockquote> <blockquote class="markdown-alert-note"> <p>Sometimes TeamViewer cannot connect to server after <em>Arch Linux</em> system upgraded, you should <strong>restart</strong> then have another try.</p> </blockquote> <h3 id="gcc-fortran">gcc-fortran</h3> <p><code>sudo pacman -S gcc-fortran</code></p> <blockquote class="markdown-alert-note"> <p>Use <em>gcc-fortran</em> instead of <em>gfortran</em> (may cause conflicts when system upgrade).</p> </blockquote> <h3 id="locate">locate</h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pacman -S mlocate</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"># Make cache before you first use it</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> updatedb</span></span></code></pre><h3 id="aria2">Aria2</h3> <p><code>sudo pacman -S aria2</code></p> <p>To speed up <code>aurman</code>:</p> <p>It’s <code>makepkg</code> but not <code>aurman</code> downloading the packages, so change the content of <code>/etc/makepkg.conf</code> to use <code>aria2</code> instead of <code>curl</code>:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>DLAGENTS=(&#39;ftp::/usr/bin/aria2c %u -o %o&#39;</span></span> <span class="giallo-l"><span> &#39;http::/usr/bin/aria2c %u -o %o&#39;</span></span> <span class="giallo-l"><span> &#39;https::/usr/bin/aria2c %u -o %o&#39;</span></span> <span class="giallo-l"><span> &#39;rsync::/usr/bin/rsync --no-motd -z %u %o&#39;</span></span> <span class="giallo-l"><span> &#39;scp::/usr/bin/scp -C %u %o&#39;)</span></span></code></pre><h3 id="r-rstudio">R &amp;&amp; RStudio</h3> <p><code>sudo pacman -S microsoft-r-open</code></p> <p><code>aurman -S rstudio-desktop-bin</code></p> <p>For RStudio server release, run <code>sudo yaourt -S rstudio-server-bin</code> instead.</p> <h3 id="steam">Steam</h3> <p>Steam outputs this error and exits.</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>symbol lookup error: /usr/lib/libxcb-dri3.so.0: undefined symbol: xcb_send_request_with_fds</span></span></code></pre> <p>For steam to work, disable dri3 in xorg config file or as a workaround run steam with <code>LIBGL_DRI3_DISABLE=1</code>. <code>LIBGL_DRI3_DISABLE=1 steam</code></p> <h3 id="jetbrains-toolbox">JetBrains Toolbox</h3> <p>Sometimes when the toolbox started after a system upgrade, the application consumes a lot CPU and displays a empty window. To solve this problem, add starting option <code>--disable-seccomp-filter-sandbox</code> in the system startup menu, then restart the application.</p> <h3 id="virtualbox">Virtualbox</h3> <p>See <a class="external" rel="noopener external" target="_blank" href="https://wiki.archlinux.org/title/VirtualBox"><em>Arch Wiki</em></a> here.</p> <h2 id="for-ubuntu">For ubuntu</h2> <h3 id="install-flash-plugin-for-web-browser">Install flash plugin for web browser</h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> apt-get install flashplugin-installer</span></span></code></pre><section class="footnotes"> <ol class="footnotes-list"> <li id="fn-1"> <p>Need 190MB at least for <em>Fedora</em>, may even less for <em>Arch Linux</em>. <a href="#fr-1-1">↩</a></p> </li> <li id="fn-2"> <p>As a server for bioinformatics use, we should use RAM first, so there’s no need of too much memory of <code>swap</code>. <a href="#fr-2-1">↩</a></p> </li> <li id="fn-3"> <p>As the disk maintaining operation system, <code>/</code> should keep all remaining space. <a href="#fr-3-1">↩</a></p> </li> </ol> </section>

Guide for say Goodbye to Windows

2017-02-15T00:00:00+00:00 2017-02-15T00:00:00+00:00 Yu Sun https://www.bigdick.live/en/blog/break-through-gfw/ <h2 id="on-digital-ocean">On <em>Digital Ocean</em></h2> <ol> <li>If you’re a student, try apply for a <a class="external" rel="noopener external" target="_blank" href="https://education.github.com/pack/">student pack</a> on <em>GitHub</em>.</li> <li>Use code of <em>Digital Ocean</em> in student pack to register a <strong>new account</strong> and buy a droplet at <em>San Fransico</em> (both node 1 &amp; 2 is OK, maybe 2 is better).</li> <li>Install <em>Fedora 25 (64 bit)</em> on your VPS then do as below:</li> </ol> <p>Install shadowsocks:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> dnf install m2crypto python-setuptools</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">easy_install</span><span class="z-string"> pip</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">pip</span><span class="z-string"> install shadowsocks</span></span></code></pre> <p>Shadowsocks use a json file as configuration. For all users, this file can be placed in <code>/etc</code>. Use <code>vi /etc/shadowsocks.json</code> to edit the file and fill in contents below:</p> <pre class="giallo z-code"><code data-lang="json"><span class="giallo-l"><span class="z-punctuation">{</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">server</span><span class="z-punctuation">&quot;:</span><span class="z-punctuation z-definition z-string z-string">&quot;0.0.0.0&quot;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">server_port</span><span class="z-punctuation">&quot;:</span><span class="z-punctuation z-definition z-string z-string">&quot;port&quot;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">local_address</span><span class="z-punctuation">&quot;:</span><span class="z-punctuation z-definition z-string z-string"> &quot;127.0.0.1&quot;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">local_port</span><span class="z-punctuation">&quot;:</span><span class="z-constant z-numeric">1080</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">password</span><span class="z-punctuation">&quot;:</span><span class="z-punctuation z-definition z-string z-string">&quot;passwd&quot;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">timeout</span><span class="z-punctuation">&quot;:</span><span class="z-constant z-numeric">300</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">method</span><span class="z-punctuation">&quot;:</span><span class="z-punctuation z-definition z-string z-string">&quot;rc4-md5&quot;</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">fast_open</span><span class="z-punctuation">&quot;:</span><span class="z-constant z-language z-json"> false</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation"> &quot;</span><span class="z-support z-type z-property-name z-json">workers</span><span class="z-punctuation">&quot;:</span><span class="z-constant z-numeric"> 1</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span></code></pre> <p>To keep shadowsocks running in background after you cut off SSH connection, you may use <code>supervisor</code>.</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> dnf install supervisor</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">echo_supervisord_conf</span><span class="z-keyword z-operator"> &gt;</span><span class="z-string"> /etc/supervisord.conf</span></span></code></pre> <p>Execute <code>vi /etc/supervisord.conf</code> to configure it:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>[program:shadowsocks]</span></span> <span class="giallo-l"><span>command=ssserver -c /etc/shadowsocks.json</span></span> <span class="giallo-l"><span>autostart=true</span></span> <span class="giallo-l"><span>autorestart=true</span></span> <span class="giallo-l"><span>user=nobody</span></span></code></pre> <p>Finally, <code>supervisord</code> or <code>supervisord -c /etc/supervisord.conf</code> to start shadowsocks server and visit <a class="external" rel="noopener external" target="_blank" href="https://github.com/shadowsocks">the project’s homepage</a> on GitHub for a client. For Windows user, try https://github.com/shadowsocks/shadowsocks-windows/releases. For Linux user, <a class="external" rel="noopener external" target="_blank" href="https://github.com/shadowsocks/shadowsocks-qt5/releases">shadowsocks-qt5</a> is a good choice. There’re also <a class="external" rel="noopener external" target="_blank" href="https://github.com/shadowsocks/shadowsocks-android/releases">shadowsocks for Android</a> and <a class="external" rel="noopener external" target="_blank" href="https://github.com/shadowsocks/shadowsocks-iOS/releases">shadowsocks for iOS</a>.</p> <h3 id="appendix">Appendix</h3> <p><code>supervisorctl status</code> for checking all processes’ status controlled by <em>supervisor</em>.</p> <p><code>supervisorctl stop ssserver</code> for kill a process.</p> <p><code>supervisorctl start ssserver</code> for start a process.</p> <p><code>supervisorctl shutdown</code> for killing all processes controlled by <em>supervisor</em>.</p> <h3 id="known-issue">Known issue</h3> <p>Sometimes shadowsocks works fine in <em>Windows</em>, but your browser can not visit pages in <em>Arch Linux</em>, it is because that too frequently updating <code>libQtShadowsocks</code> brings conflict.</p> <p>To <a class="external" rel="noopener external" target="_blank" href="https://github.com/shadowsocks/shadowsocks-qt5/issues/550">solve this problem</a>, you can downgrade <code>libQtShadowsocks</code> or change the encryption method from <code>AES-256-CFB</code> to <code>AES-256-CTR</code>.</p> <h2 id="on-aws">On <em>AWS</em></h2> <ol> <li> <p>Creat an <em>EC2</em> (choose <em>Ubuntu</em> here) instance, and be sure to keep in mind that the key pair should be generated during the process, then download your private key (.pem) file.</p> </li> <li> <p>Change your private key’s priviledge to 400: <code>chmod 400 /path/my-key-pair.pem</code>.</p> </li> <li> <p>Log-in your instance with such command: <code>ssh -i /path/my-key-pair.pem ubuntu@ec2-13-124-234-100.ap-northeast-2.compute.amazonaws.com</code>.</p> </li> <li> <p>Upgrade OS:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> apt-get update</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> apt-get dist-upgrade</span></span></code></pre></li> <li> <p>Install Shadowsocks:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> apt-get install python-pip</span></span> <span class="giallo-l"><span class="z-entity z-name z-function">sudo</span><span class="z-string"> pip install shadowsocks</span></span></code></pre></li> </ol> <blockquote class="markdown-alert-note"> <p>You may see this error during installation of Shadowsocks:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>Traceback (most recent call last):</span></span> <span class="giallo-l"><span>File &quot;/usr/bin/pip&quot;, line 11, in &lt;module&gt;</span></span> <span class="giallo-l"><span> sys.exit(main())</span></span> <span class="giallo-l"><span>File &quot;/usr/lib/python2.7/dist-packages/pip/__init__.py&quot;, line 215, in main</span></span> <span class="giallo-l"><span> locale.setlocale(locale.LC_ALL, &#39;&#39;)</span></span> <span class="giallo-l"><span>File &quot;/usr/lib/python2.7/locale.py&quot;, line 581, in setlocale</span></span> <span class="giallo-l"><span> return _setlocale(category, locale)</span></span> <span class="giallo-l"><span>locale.Error: unsupported locale setting</span></span></code></pre> <p>It is caused by wrong language settings. Fix it by <code>export LC_ALL=C</code> (only works in <strong>current</strong> tty), then try again.</p> </blockquote> <ol start="6"> <li>Start Shadowsocks server by: <code>sudo ssserver -p 2333 -k password -m rc4-md5 -d start</code> (use <code>2333</code> as port and <code>password</code> as password, change <code>start</code> to <code>stop</code> if you wanna stop the process).</li> </ol> <blockquote class="markdown-alert-note"> <p>You should use <code>AES-256-CTR</code> as encryption method at ArchLinux to avoid problems brought by over-updated <code>libqtshadowsocks</code>.</p> </blockquote> <h3 id="optimization">Optimization</h3> <h4 id="break-though-limit-of-file-open-number">Break though limit of file open number</h4> <ol> <li> <p>Edit <code>limits.conf</code>: <code>sudo vi /etc/security/limits.conf</code>.</p> </li> <li> <p>Add those two lines:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>* soft nofile 51200</span></span> <span class="giallo-l"><span>* hard nofile 51200</span></span></code></pre></li> <li> <p>Restart the OS.</p> </li> </ol> <h4 id="improve-system-performance">Improve system performance</h4> <ol> <li> <p>Edit <code>sysctl.conf</code>: <code>sudo vi /etc/sysctl.conf</code>.</p> </li> <li> <p>Add these lines:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>fs.file-max = 51200</span></span> <span class="giallo-l"><span>net.core.rmem_max = 67108864</span></span> <span class="giallo-l"><span>net.core.wmem_max = 67108864</span></span> <span class="giallo-l"><span>net.core.netdev_max_backlog = 250000</span></span> <span class="giallo-l"><span>net.core.somaxconn = 4096</span></span> <span class="giallo-l"><span>net.ipv4.tcp_syncookies = 1</span></span> <span class="giallo-l"><span>net.ipv4.tcp_tw_reuse = 1</span></span> <span class="giallo-l"><span>net.ipv4.tcp_tw_recycle = 0</span></span> <span class="giallo-l"><span>net.ipv4.tcp_fin_timeout = 30</span></span> <span class="giallo-l"><span>net.ipv4.tcp_keepalive_time = 1200</span></span> <span class="giallo-l"><span>net.ipv4.ip_local_port_range = 10000 65000</span></span> <span class="giallo-l"><span>net.ipv4.tcp_max_syn_backlog = 8192</span></span> <span class="giallo-l"><span>net.ipv4.tcp_max_tw_buckets = 5000</span></span> <span class="giallo-l"><span>net.ipv4.tcp_fastopen = 3</span></span> <span class="giallo-l"><span>net.ipv4.tcp_rmem = 4096 87380 67108864</span></span> <span class="giallo-l"><span>net.ipv4.tcp_wmem = 4096 65536 67108864</span></span> <span class="giallo-l"><span>net.ipv4.tcp_mtu_probing = 1</span></span> <span class="giallo-l"><span>net.ipv4.tcp_congestion_control = hybla</span></span></code></pre></li> <li> <p>Run <code>sudo sysctl -p</code> to reload settings.</p> </li> </ol> <h4 id="serverspeeder">ServerSpeeder</h4> <p>ServerSpeeder cannot be used on Ubuntu 16.04 now.</p> <h4 id="net-speeder">net-speeder</h4> <p>net-speeder will double the network traffic. Not recommended.</p>

Guide to break through Chinese government's GFW

2017-02-09T00:00:00+00:00 2017-02-09T00:00:00+00:00 Yu Sun https://www.bigdick.live/en/blog/perl-oneliners/ <p>To use most of one-liners in this post, you should:</p> <ol> <li>If you’re working on <em>Windows</em>, download and install <a class="external" rel="noopener external" target="_blank" href="https://downloads.activestate.com/">ActiveState Perl</a>.</li> <li>Replace variable <code>$file</code> at the end of scripts by the real file name.</li> <li>Run the script as a command in <code>CMD</code>.</li> </ol> <ul> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#extract-a-reference-sequence-from-a-single-query-fasta-file">Extract a reference sequence from a single-query <em>FASTA</em> file</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#find-plain-text-files-with-same-appendix-in-multiple-directories-recursively-and-merge-them-into-one-file">Find plain-text files with same appendix in multiple directories recursively and merge them into one file</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#filter-fastq-by-length-of-query">Filter <em>FASTQ</em> by length of query</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#to-flare-out-fasta-queries">To “flare out” <em>FASTA</em> queries</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#to-calculate-the-proportion-of-gaps-n-or-n-in-a-genome">To calculate the proportion of gaps (“N” or “n”) in a genome</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#to-get-reads-in-fastq-with-specific-length-and-count-in-descending-order">To get reads in <em>FASTQ</em> with specific length and count in descending order</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#to-calculate-positive-and-negtive-read-counts-separately">To calculate positive and negtive read counts separately</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#to-print-file-names-in-current-directory">To print file names in current directory</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#to-remove-all-patches-and-scaffolds-etc-but-leave-only-chromosomes-in-a-genome-reference-file">To remove all patches and scaffolds etc., but leave only chromosomes in a genome reference file.</a></li> <li><a href="https://www.bigdick.live/en/blog/perl-oneliners/#to-convert-fastq-into-fasta">To convert <em>FASTQ</em> into <em>FASTA</em></a></li> </ul> <h2 id="extract-a-reference-sequence-from-a-single-query-fasta-file">Extract a reference sequence from a single-query <em>FASTA</em> file</h2> <p>Replace the scalar $start and $end with two number indicating location then execute.</p> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -</span><span class="z-support z-function">ne</span><span class="z-punctuation z-definition z-string z-string"> &#39;next if /^&gt;/; $_ =~ s/[\r\n]+//g; @a = split //, $_; for (@a) {$i ++; if(($i &gt;= $start) &amp;&amp; ($i &lt;= $end)) {print}}&#39;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre><h2 id="find-plain-text-files-with-same-appendix-in-multiple-directories-recursively-and-merge-them-into-one-file">Find plain-text files with same appendix in multiple directories recursively and merge them into one file</h2> <p>Replace the scalar $append with the suffix (e.g. .seq) then execute.</p> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -MFile::Find -</span><span class="z-support z-function">le</span><span class="z-punctuation z-definition z-string z-string"> &quot;sub search{if (/</span><span class="z-constant z-character z-escape">\$</span><span class="z-string">append</span><span class="z-punctuation z-definition z-variable z-variable">$/</span><span class="z-string">){(</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$id</span><span class="z-string">) = map{/.+_(</span><span class="z-constant z-character z-escape">\(</span><span class="z-string">.+).seq</span><span class="z-punctuation z-definition z-variable z-variable">$/</span><span class="z-string">} </span><span class="z-punctuation z-definition z-variable z-variable">$_</span><span class="z-string">; open(SEQ, qq{&lt;}, </span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$File::Find::name</span><span class="z-string">) or die qq{Processing:</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$File::Find::name</span><span class="z-constant z-character z-escape">\n</span><span class="z-punctuation z-definition z-variable z-variable">$!</span><span class="z-constant z-character z-escape">\n</span><span class="z-string">}; while(&lt;SEQ&gt;) {print MERGE qq{&gt;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$id</span><span class="z-string">}; print MERGE </span><span class="z-punctuation z-definition z-variable z-variable">$_</span><span class="z-string">} }} open MERGE, qq{&gt;}, qq{sanger.fa}; find({ wanted =&gt; </span><span class="z-constant z-character z-escape">\&amp;</span><span class="z-string z-punctuation z-definition z-string">search, no_chdir =&gt; 1 }, qq{.})&quot;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre><h2 id="filter-fastq-by-length-of-query">Filter <em>FASTQ</em> by length of query</h2> <p>Replace the scalar $minlen and $maxlen with the number representing interval then execute.</p> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -lane </span><span class="z-punctuation z-definition z-string z-string">&quot;if (</span><span class="z-punctuation z-definition z-variable z-variable">$.</span><span class="z-string"> % 4 == 1) {</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$id</span><span class="z-string"> = </span><span class="z-punctuation z-definition z-variable z-variable">$_</span><span class="z-string"> =~ s/[</span><span class="z-constant z-character z-escape">\r\n</span><span class="z-string">]+//r} elsif (</span><span class="z-punctuation z-definition z-variable z-variable">$.</span><span class="z-string"> % 4 == 2) {</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$seq</span><span class="z-string"> = </span><span class="z-punctuation z-definition z-variable z-variable">$_</span><span class="z-string"> =~ s/[</span><span class="z-constant z-character z-escape">\r\n</span><span class="z-string">]+//r} elsif (</span><span class="z-punctuation z-definition z-variable z-variable">$.</span><span class="z-string"> % 4 == 3) {</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$name</span><span class="z-string"> = </span><span class="z-punctuation z-definition z-variable z-variable">$_</span><span class="z-string"> =~ s/[</span><span class="z-constant z-character z-escape">\r\n</span><span class="z-string">]+//r} else {</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$qual</span><span class="z-string"> = </span><span class="z-punctuation z-definition z-variable z-variable">$_</span><span class="z-string"> =~ s/[</span><span class="z-constant z-character z-escape">\r\n</span><span class="z-string">]+//r; print qq{</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$id</span><span class="z-constant z-character z-escape">\n</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$seq</span><span class="z-constant z-character z-escape">\n</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$name</span><span class="z-constant z-character z-escape">\n</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$qual</span><span class="z-string">} if ((length(</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$seq</span><span class="z-string">) &gt;= </span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$minlen</span><span class="z-string">) &amp;&amp; (length(</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$seq</span><span class="z-string">) &lt;= </span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite">$maxlen</span><span class="z-string z-punctuation z-definition z-string">))}&quot;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre><h2 id="to-flare-out-fasta-queries">To “flare out” <em>FASTA</em> queries</h2> <p><em>FASTA</em> is a file format and is recommended that all lines of text be shorter than 80 characters. However, when we need to grep continuous strings in it, the line break such as <code>[\r\n]</code> usually cause troubles. Use this command can remove newline characters in sequences.</p> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -pe </span><span class="z-punctuation z-definition z-string z-string">&#39;$. == 1 ? 23333333 : /^&gt;/ ? $_ = qq{\n} . $_ : s/[\r\n]+//&#39;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre><h2 id="to-calculate-the-proportion-of-gaps-n-or-n-in-a-genome">To calculate the proportion of gaps (“N” or “n”) in a genome</h2> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -lne </span><span class="z-punctuation z-definition z-string z-string">&#39;next if /^&gt;/; $_ =~ s/[\r\n]+//; $total_len += length($_); $n_count += map{/[nN]/g} $_ }{ END{print qq{Total length: $total_len}; print qq{N count: $n_count}; $n_ratio = $n_count / $total_len; print qq{N ratio: $n_ratio}}&#39;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre><h2 id="to-get-reads-in-fastq-with-specific-length-and-count-in-descending-order">To get reads in <em>FASTQ</em> with specific length and count in descending order</h2> <p>Replace the scalar <code>$length</code> with the length you’re interested in (without unit such as “bp”) then execute.</p> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -lne </span><span class="z-punctuation z-definition z-string z-string">&#39;next unless (($. % 4 == 2)&amp;&amp;(length($_) == $length)); $count{$_}++; }{ for (sort {$count{$b} &lt;=&gt; $count{$a}} keys %count) {print qq{$_\t$count{$_}}}&#39;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre><h2 id="to-calculate-positive-and-negtive-read-counts-separately">To calculate positive and negtive read counts separately</h2> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -lane </span><span class="z-punctuation z-definition z-string z-string">&#39;next if /^@/; next if $F[1]&amp;0x4; $len = eval join q{+}, map {/(\d+)[MID]/g} $F[5]; $pos{$len} ++ if $F[1] == 0; $neg{$len} ++ if $F[1] == 16 }{ print qq{$_\t$pos{$_}} for (keys %pos); print qq{$_\t-$neg{$_}} for (keys %neg)&#39;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre><h2 id="to-print-file-names-in-current-directory">To print file names in current directory</h2> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -</span><span class="z-support z-function">le</span><span class="z-punctuation z-definition z-string z-string"> &quot;print for (glob qq{*})&quot;</span></span></code></pre><h2 id="to-remove-all-patches-and-scaffolds-etc-but-leave-only-chromosomes-in-a-genome-reference-file">To remove all patches and scaffolds etc., but leave only chromosomes in a genome reference file.</h2> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -lpe </span><span class="z-punctuation z-definition z-string z-string">&#39;/^&gt;/ &amp;&amp; !/REF$/ ? last : /^&gt;/ ? $_ =~ s/&gt;(\w+)/&gt;chr$1/ : 23333333&#39;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre><h2 id="to-convert-fastq-into-fasta">To convert <em>FASTQ</em> into <em>FASTA</em></h2> <pre class="giallo z-code"><code data-lang="perl"><span class="giallo-l"><span class="z-source">perl -F</span><span class="z-punctuation z-definition z-string z-string">&#39;@&#39;</span><span class="z-source"> -lanE</span><span class="z-punctuation z-definition z-string z-string">&#39;$. % 4 == 1 ? say qq{&gt;$F[1]} : $. % 4 == 2 ? say $_ : $. % 4 == 3 ? qq{Yu Sun is handsome} : 233333333&#39;</span><span class="z-punctuation z-definition z-variable z-variable z-other z-readwrite"> $file</span></span></code></pre>

Here I place some Perl-oneliners (may be not common used) by me. You can use them to modify, count , summarize or change file type on plain-text (like fastx).

2016-05-21T00:00:00+00:00 2016-05-21T00:00:00+00:00 Yu Sun https://www.bigdick.live/en/projects/dota2-automation-scripts/ <p>These are two Dota2 automation scripts from 2016, preserved as they were rather than presented as maintained tools.</p> <h2 id="contents"><a class="header-anchor no-hover-padding" href="#contents" aria-label="Anchor link for: contents"><span class="link-icon" aria-hidden="true"></span></a> Contents</h2> <ul> <li><a class="external" rel="noopener external" target="_blank" href="https://github.com/bioinformatist/bioinformatist.github.io/blob/main/static/archives/legacy-dota2-automation/dota2_lumbering_challenge.py">Dota2 lumbering challenge auto-clicker</a>: a Windows / Python script that listens for <code>Ctrl+C</code> and randomly clicks inside a fixed screen area for <code>60</code> seconds.</li> <li><a class="external" rel="noopener external" target="_blank" href="https://github.com/bioinformatist/bioinformatist.github.io/blob/main/static/archives/legacy-dota2-automation/auto_spectate.sh">Dota2 Source 2 auto-spectating script</a>: a Linux / X11 script that uses <code>xdotool</code> to activate the Dota2 window and periodically click the spectating entry point.</li> </ul> <h2 id="lumbering-challenge"><a class="header-anchor no-hover-padding" href="#lumbering-challenge" aria-label="Anchor link for: lumbering-challenge"><span class="link-icon" aria-hidden="true"></span></a> Lumbering Challenge</h2> <p><img src="https://www.bigdick.live/en/projects/dota2-automation-scripts/chopping.gif" alt="Auto-clicking demo" /></p> <p>The idea was simple: trigger from the keyboard, then simulate mouse clicks in the game area.</p> <p><img src="https://www.bigdick.live/en/projects/dota2-automation-scripts/chopping_result.png" alt="Lumbering result" /></p>

Two Dota2 automation scripts from 2016: lumbering challenge auto-clicking and Source 2 auto-spectating.