[{"content":"Geçtiğimiz günlerde Gophers İstanbul 2026\u0026lsquo;da bir sunum yaptım. Konu: tek bir Go binary\u0026rsquo;si ile Docker container\u0026rsquo;ları, libvirt/KVM sanal makineleri, DHCP+DNS altyapısını ve WireGuard VPN tünelini — hepsini bir YAML dosyasından ayağa kaldıran bir araç. Sunumda bol bol kod parçacıkları ve Go\u0026rsquo;nun iç yapısı var. Bu yazı o değil. Burada daha çok aracın arkasındaki dağınık gerçek-dünya probleminden, nerelerde gerçekten işe yaradığını düşündüğümden, nelerin hâlâ eksik olduğundan ve vakit bulursam denemek istediğim fikirlerden bahsetmek istiyorum.\n📊 Sunum slaytları · 📊 Slides (English) · 💻 Kaynak kod\nProblem Bir noktada çoğu backend ya da platform ekibi benzer bir duvara tosluyor: aynı makine üzerinde hem container hem de VM çalıştırmanız gereken bir demo ya da test ortamı ihtiyacı çıkıyor. İlk başta kulağa çok büyük bir mesele gibi gelmiyor. “Bir iki script yazarız, ayağa kalkar” diye düşünüyorsunuz. Sonra işin detayları yavaş yavaş ortaya çıkıyor.\nÖnce Docker ağı için küçük bir script yazılıyor. Ardından DHCP tarafı için bir konfigürasyon üreten başka bir script ekleniyor. Sonra virsh için XML tanımları geliyor. Her şeyin hazır olduğunu düşündüğünüz noktada bu kez VM’in IP almadığını fark ediyorsunuz. Biraz kurcalayınca sorunun bridge arayüzünde olduğunu görüyorsunuz; promiscuous mode açık değil. Bu defa zincire bir ip link set daha ekleniyor.\nAsıl sorun ise genelde daha sonra ortaya çıkıyor. Birisi sizden aynı ortamı başka bir makinede tekrar kurmanızı istediğinde, yazdığınız şeyin önemli bir kısmının fark etmeden kendi geliştirme makinenize bağlandığını anlıyorsunuz. Interface isimleri, dosya yolları, varsayımlar, geçici çözümler… Her şey ilk kurulduğu anda mantıklı görünse de tekrar üretilebilirlik tarafında tablo pek iç açıcı olmuyor.\nBen yıllar önce iş yerinde tam olarak bu problemle karşılaştım. İhtiyacımız olan şey izole hibrit ortamlardı. Bazı servisler container içinde sorunsuz çalışıyordu, ama bazı iş yükleri gerçekten sanal makine gerektiriyordu. Özellikle kendi kernel’ine ihtiyaç duyan ya da container içinde simüle edilmesi anlamsız olan bileşenlerde VM kaçınılmazdı.\nSorun şu ki manuel kurulum hem yavaştı hem de güvenilmezdi. Ortamı baştan kurmak genelde 15-20 dakika sürüyordu ve neredeyse hiçbir zaman tamamen aynı sonucu vermiyordu. Bazen bir container geride kalıyor, bazen bridge ağı yeniden başlatmalardan sonra yaşamaya devam ediyor, bazen de eski VM tanımları yeni çalıştırmalarla çakışıyordu. Ortamı ayağa kaldırmak can sıkıcıydı ama kaldırmak çoğu zaman daha da kötüydü.\nBir süre sonra bunun script koleksiyonuyla çözülecek bir problem olmadığını fark ettim. Parça parça otomasyon eklemek yerine, ortamın tamamını yöneten tek bir araç olması çok daha mantıklıydı. Bunun üzerine her şeyi tek yerden yöneten küçük bir orchestrator yazdım.\nKullanımı mümkün olduğunca basit tuttum:\norchestrator up -c config.yaml # ortamı ayağa kaldır orchestrator status # ne çalışıyor kontrol et orchestrator ssh demo-vm # VM\u0026#39;e SSH ile bağlan orchestrator down # ortamı temiz şekilde kaldır Amaç yalnızca “çalıştırmak” değildi. Aynı zamanda tekrar üretilebilir, taşınabilir ve temiz kaldırılabilir bir yapı kurmaktı. Bir ortam ayağa kalktığında hangi ağların oluşturulduğunu, hangi VM’lerin tanımlandığını, hangi container’ların bağlı olduğunu sistematik biçimde bilmek; iş bittiğinde de bunların hepsini geride iz bırakmadan temizleyebilmek gerekiyordu.\nAraçla ilgili sevdiğim detaylardan biri de dağıtım tarafının çok sade olması oldu. Uygulama yaklaşık 15 MB’lık statik bir binary olarak derleniyor. Herhangi bir çalışma zamanı bağımlılığı gerektirmiyor. Dosyayı sunucuya scp ile kopyalayıp doğrudan çalıştırabiliyorsunuz. Özellikle demo ortamları, geçici lab kurulumları ya da farklı makinelerde hızlı tekrar kurulum gereken senaryolarda bu sadelik ciddi fark yaratıyor.\nBu araç ortaya çıkarken asıl hedefim “yeni bir platform” yazmak değildi. Daha çok, dağınık bash script’leri, unutulan ağ ayarlarını ve her makinede farklı davranan el yapımı kurulumları tek bir akışta toplamak istedim. Sonuçta ortaya çıkan şey, bizim ekip için hem kurulum süresini düşüren hem de hata ayıklamayı ciddi biçimde kolaylaştıran bir çözüm oldu.\nBazen en faydalı araçlar en iddialı olanlar değil, her gün tekrar edilen can sıkıcı işleri ortadan kaldıranlar oluyor. Bu da benim için tam olarak öyle bir projeydi.\nGerçekte Ne Yapıyor? Aracın yaptığı işi en basit haliyle şöyle özetleyebilirim: tek bir sunucu üzerinde, tek bir binary ve tek bir config dosyasıyla hibrit bir ortam kuruyor.\norchestrator up çalıştığında arka planda birkaç adımı sırayla yürütüyor. Önce Docker tarafında gerekli bridge ağı oluşturuyor. Ardından DHCP ve DNS işini üstlenecek container’ları başlatıyor. Sonrasında uygulama container’larıyla birlikte VM’leri ayağa kaldırıyor. Bu aşamada işleri mümkün olduğunca paralel yürüttüğü için, her şey tek tek beklenerek değil eşzamanlı biçimde başlıyor. Kurulum tamamlanırken uzak erişim için WireGuard istemci yapılandırmasını üretiyor ve en son, daha sonra orchestrator down çağrıldığında neyin nasıl geri alınacağını bilmek için bir durum dosyası yazıyor.\nAslında burada çözmeye çalıştığı problem oldukça net. Docker size namespace’lerle izole edilmiş container’lar veriyor. Libvirt ise kendi kernel’ine sahip, tam anlamıyla ayrı makineler gibi davranan sanal makineler sunuyor. Tek başına bakınca iki dünya da tanıdık. Zor olan kısım, bunları aynı ağ düzleminde düzgün şekilde bir araya getirmek.\nBu aracın esas işi de tam olarak burada başlıyor: container’ları ve VM’leri aynı Layer 2 bridge üzerinde buluşturmak, üstelik ikisinin de aynı DHCP akışından IP almasını sağlamak. Böylece ortam dışarıdan bakıldığında iki ayrı altyapının yamalı birleşimi gibi değil, tek parça bir sistem gibi davranıyor.\nBunun pratikteki karşılığı önemli. Uzak bir istemci WireGuard üzerinden ortama bağlandığında, karşısında “bu container ağı, şu VM ağı” gibi ayrımlar görmüyor. Servislere, arkada container mı var VM mi diye düşünmeden, şeffaf biçimde erişebiliyor. Bence aracın gerçek değeri de burada: farklı çalışma modellerini tek bir ağ ve yaşam döngüsü altında toplamak.\nGerçekten Nerelerde İşe Yarar? Baştan net söyleyeyim: bu araç bir Kubernetes alternatifi değil. Zaten öyle olmak için de tasarlanmadı. Bilinçli olarak küçük tutulmuş, ne yaptığını bilen ve belirli senaryolarda gerçekten iş gören bir araç.\nHer yere uysun diye şişirilmiş bir platform olmaktan çok, “tek makinede container + VM karışık bir ortamı hızlı, tekrar üretilebilir ve temiz şekilde ayağa kaldırayım” problemine odaklanıyor. Bu yüzden en çok şu tip durumlarda anlamlı hale geliyor:\nEğitim ve Workshop Ortamları İki günlük bir güvenlik workshop’u düzenlediğinizi düşünün. Her katılımcının kendi izole ağı var. O ağın içinde bir web uygulaması container olarak çalışıyor, yanında kurcalanacak savunmasız bir VM bulunuyor, bir de içerideki servis isimlerini çözen DNS gerekiyor.\nBunu klasik yöntemlerle yapmak genelde iki yere çıkıyor: ya bulut tarafında kişi başı VPC benzeri yapılar açıyorsunuz ve maliyet hızla büyüyor, ya da herkese Vagrant benzeri bir kurulum verip “umarım laptop kaldırır” noktasına geliyorsunuz. Teoride güzel, pratikte sabahın ilk yarısı kurulum sorunlarıyla geçiyor.\nBu araçta yaklaşım daha sade. Güçlü tek bir paylaşımlı sunucu üzerinde her öğrenciye kendi YAML tanımı ve kendi WireGuard tüneli veriliyor. Herkes aynı fiziksel kaynak üzerinde, ama birbirinden izole ortamlarda çalışıyor. Eğitmen tarafında da yönetmesi çok daha kolay oluyor.\nEdge / IoT Prototipleme Edge ya da IoT tarafında prototip geliştirirken ihtiyaçlar genelde biraz garip oluyor. Bazı parçalar container içinde rahatça çalışıyor, ama firmware testi ya da daha düşük seviye sistem davranışları için tam bir Linux VM gerekiyor. Böyle bir durumda kimsenin ilk refleksi “buraya bir Kubernetes kuralım” olmuyor.\nİstenen şey genelde çok daha basit: binary’yi cihaza at, config dosyasını bırak, tek komutla ortamı ayağa kaldır.\nGOOS=linux GOARCH=arm64 go build -o orchestrator . Cross-compile tarafının kolay olması da burada ciddi avantaj. Özellikle ARM tabanlı edge cihazlarda, ekstra paket yöneticisi ya da karmaşık runtime bağımlılıklarıyla uğraşmadan doğrudan çalıştırabilmek büyük rahatlık.\nVM Gerektiren CI Entegrasyon Testleri Bazı şeyler gerçekten container içinde doğru dürüst test edilemiyor. Kernel modülleri, özel ağ davranışları, gerçek boot sırası bekleyen servisler ya da sistemin gerçekten “makine gibi” davranmasını gerektiren senaryolar bunların başında geliyor.\nBugün birçok CI pipeline’ında bu testler ya tamamen pas geçiliyor ya da pahalı ve kırılgan nested virtualization çözümleriyle yürütülüyor. İkisi de çok iç açıcı değil.\nBu tür bir araç ise pipeline’ın başında hibrit ortamı kurup iş bitince düzgünce kaldırmak için uygun bir model sunuyor. Özellikle up ve down tarafının idempotent olacak şekilde düşünülmesi burada önemli; çünkü CI dünyasında aynı şeyi tekrar tekrar, temiz biçimde yapabilmek her şeyden daha değerli.\nHızlı Demo Ortamları Satış mühendisleri, çözüm mimarları, destek ekipleri… Kısacası bir ürünü ya da sistemi “çalışır halde” göstermek zorunda olan herkes benzer bir dert yaşıyor. Demo ortamı genelde bir noktada kırılgan bir kar topuna dönüşüyor. Bir VM imajı var, onun üstünde elle yapılmış ayarlar var, kimsenin tam hatırlamadığı bir network workaround’u var; dokununca bozuluyor ama dokunmadan da yaşlanıyor.\nBunun yerine ortamı versiyon kontrollü bir YAML dosyasında tarif etmek çok daha temiz bir yaklaşım. Gerektiğinde birkaç saniye içinde yeniden kurabiliyorsunuz. Demo makinesine “aman buna dokunmayın” muamelesi yapmak yerine, gerektiğinde sıfırdan üretilebilen bir yapıya geçmiş oluyorsunuz.\nHomelab Orkestrasyonu Bir de işin homelab tarafı var. Evde bare-metal sunucu ya da küçük bir Proxmox kutusu çalıştıranlar bu fikre hemen yakın hissedebilir. Bazen tek ihtiyacınız, tek node üzerinde tekrar kurulabilir container + VM yığınlarıdır. Ama bunun için tam boy bir hyperconverged platformun bütün ağırlığını da sırtlanmak istemezsiniz.\nBu durumda daha hafif, doğrudan ve anlaşılır bir araç daha mantıklı olabiliyor. Özellikle “tek makine, birkaç servis, bir iki VM, düzgün ağ izolasyonu ve temiz lifecycle yönetimi” çizgisinde kalıyorsanız, küçük bir çözüm çoğu zaman büyük platformlardan daha kullanışlı oluyor.\nNasıl Kullanılır? Bu aracın bütün mantığı config dosyasında yaşıyor. İşin güzel tarafı da burada: ortamı kuran şey dağınık script’ler değil, tek bir yerde duran açık bir tanım.\nnetwork_name: demo-net subnet: 172.19.5.0/24 network_type: bridge containers: - name: web-demo image: nginx:alpine ip: 172.19.5.10 - name: whoami image: containous/whoami:latest ip: 172.19.5.11 vms: - name: demo-vm image: ./images/debian-12.qcow2 memory_mb: 1024 vcpus: 2 packages: [curl, wget, vim, htop, net-tools, nmap] wireguard: enabled: true peer_name: demo-client address: 10.10.0.2/24 Buradaki fikir basit: ortamın neye benzediğini tek tek komutlarla değil, deklaratif bir dosyayla tarif ediyorsunuz. Ağ adı ne, subnet ne, hangi container’lar çalışacak, hangi VM açılacak, içine hangi paketler yüklenecek, WireGuard olacak mı — hepsi burada.\nBu dosya bir bakıma “kurulum notu” değil, doğrudan ortamın kendisi.\nEn iyi şekilde kullanmak için birkaç pratik not:\nConfig’i versiyonlayın, ortamı değil.\nBence işin en kritik noktası bu. Ortamı kafada ya da bir wiki sayfasında tutmaya çalışmak yerine config dosyasını Git’e koyuyorsunuz. Sonra biri gelip “geçen çeyrekte müşteriye gösterdiğimiz demo tam olarak nasıldı?” diye sorduğunda, kimsenin hafızasına güvenmiyorsunuz.\ngit log git show \u0026lt;commit\u0026gt; Adres referanslanan servislerde statik IP kullanın.\nBaşka servislerin bağımlı olduğu bileşenler — mesela web frontend, API gateway ya da dahili DNS gibi şeyler — sabit IP ile tanımlanmalı. Böylece “o servis bugün hangi IP’yi aldı?” diye dolaşmanız gerekmiyor. Ama gerçekten önemi olmayan, kimsenin doğrudan adreslemediği worker benzeri işler için dinamik tahsis gayet yeterli.\nVM tarafında paketleri baştan hazırlamak ciddi zaman kazandırıyor.\npackages alanı boşuna yok. Araç bunu görüp ilk boot sırasında gerekli paketleri cloud-init üzerinden yüklüyor. Yani VM açıldığında curl, wget, nmap, vim gibi temel araçlar zaten hazır oluyor. Özellikle internet erişimi sınırlı ortamlarda ya da “bu makine dışarı çıkamayacak” senaryolarında bu küçük detay ciddi fark yaratıyor.\nWireGuard yapılandırması doğrudan kullanılabilir dosya olarak üretiliyor.\norchestrator up tamamlandıktan sonra elinizde istemciye verebileceğiniz hazır bir konfigürasyon oluyor. İnsanların tek tek anahtar üretmesi, peer eklemesi, “şu public key’i bana atsana” diye mesajlaşması gerekmiyor.\norchestrator up -c config.yaml ls wg-client-*.conf Dosyayı import ediyorlar ve ortama bağlanıyorlar. En sevdiğim kullanım türlerinden biri bu, çünkü gerçekten sürtünmeyi azaltıyor.\nNeleri Eksik? (Dürüst Olmak Gerekirse) Konferans sunumunda tabii ki daha parlak taraflarını gösterdim. Ama işin dürüst kısmı şu: araç kullanışlı olsa da bazı yerleri hâlâ köşeli. Hatta bazı eksikleri özellikle not etmek istiyorum, çünkü bunlar gerçek dünyada bir noktada karşınıza çıkıyor.\nDüzgün Kapanış Zinciri Yok Şu anda orchestrator down çağrıldığında ortam kapanıyor, kaynaklar temizleniyor, işler toparlanıyor. Ama bütün bunlar bilinçli bir bağımlılık sırasıyla yapılmıyor.\norchestrator down Bu şu anlama geliyor: eğer bir VM’in düzgün kapanmadan önce ağdaki başka bir servise ulaşması gerekiyorsa ya da veri flush etmesi için belli bir bileşenin biraz daha ayakta kalması şartsa, sistem bunu anlamıyor. Kısacası “önce bunu kapat, sonra şunu indir” gibi akıllı bir kapanış grafiği henüz yok.\nYapı taşları aslında var. Context iptali var, WaitGroup mantığı var, eşzamanlı işlemler kontrol altında. Ama bağımlılık sırasını yöneten gerçek bir shutdown orchestration katmanı henüz eklenmiş değil. Muhtemelen doğru yaklaşım, başlatma sırasının tersini uygulayan bir DAG mantığı olurdu.\nYalnızca Tek Sunucu Bu araç en başından beri tek sunucu varsayımıyla yazıldı. Zaten tasarım hedefi de buydu: bir makine, bir binary, bir config, tek komut.\nAma bunun doğal sınırı şu: VM’leri farklı sunuculara dağıtmak, ağı federasyon yapmak, işleri cluster mantığıyla yaymak gibi kavramlar yok. Eğitim ortamı, hızlı demo, küçük CI senaryoları ve homelab için bu çoğu zaman sorun değil. Hatta sadelik sağlıyor. Ama ölçeği büyütmek istediğiniz anda sınır çizgisi gayet netleşiyor.\n\u0026ldquo;Çalışıyor mu?\u0026rdquo; Ötesinde Sağlık Kontrolü Yok Bugünkü sağlık kontrolü temelde şuna bakıyor: container ya da VM ayağa kalktı mı, süreç running durumda mı?\nYani sistem “çalışıyor” ile “işini doğru yapıyor” arasındaki farkı henüz bilmiyor.\nMesela nginx container’ı açılmış olabilir ama yanlış config yüzünden 502 dönüyordur. Ya da servis portu açılmıştır ama uygulama içeride kendine gelmemiştir. Bu durumda orchestrator status size kötü bir haber vermez; dışarıdan bakınca her şey yolundaymış gibi görünür.\norchestrator status HTTP probe, TCP check, readiness gate ya da uygulama seviyesinde health validation gibi şeyler henüz yok. Açık söylemek gerekirse, bu eklendiği anda araç bir üst seviyeye çıkmış olacak.\nVM Ağ Yapılandırması Kırılgan\nContainer’ları aynı bridge’e almak kolay kısım. VM’i aynı yapıya düzgün bağlamak ise hâlâ biraz “burası Linux networking, burada kimse masum değil” kategorisinde.\nBridge’in promiscuous modda olması gerekiyor, bağlantının gerçekten oluştuğunu doğrulamak gerekiyor, bridge helper doğru yapılandırılmış olmalı. Araç bunları hallediyor ama işin altında yine biraz hassas sistem davranışı yatıyor. Bir kernel güncellemesi, host tarafında ufak bir fark ya da dağıtıma özel bir değişiklik bu zinciri bozabilir.\nip link set \u0026lt;bridge\u0026gt; promisc on brctl show Daha sağlam bir yaklaşım belki macvlan ya da ayrılmış bir OVS bridge olurdu. Şu anki çözüm çalışıyor, ama “bunu sonsuza kadar düşünmeden bırakırım” türden değil.\nÇok Kiracılık (Multi-Tenancy) Yok Şu an aynı sunucuda iki farklı kullanıcının birbirinden bağımsız biçimde orchestrator up çalıştırdığını düşünün. Eğer isimlendirme ve ağ tarafı dikkatli tasarlanmamışsa, çakışmalar hemen başlıyor.\norchestrator up -c ata.yaml orchestrator up -c turkmen.yaml Container adları çakışabilir, network isimleri üst üste binebilir, port binding’ler birbirine girebilir. Yani mevcut haliyle bu araç “çok kullanıcılı lab platformu” olmaktan çok, kontrollü biçimde kullanılan tek operatörlü bir araç.\nKullanıcı ya da oturum bazlı namespace mantığı eklense — isim önekleri, izole alt ağlar, session ID tabanlı kaynak adları gibi — özellikle workshop tarafında çok daha rahat kullanılabilir hale gelirdi.\nDurum Yönetimi İlkel Durum dosyası var, ama şu anda oldukça düz bir model. Temelde JSON halinde “ne oluşturuldu” bilgisini yazıyor ve down sırasında buna güveniyor.\nBu basitlik belli bir noktaya kadar güzel. Ama binary provisioning sırasında çökerse ya da host üstünde dışarıdan bir değişiklik olursa, durum dosyasıyla gerçek dünya birbirinden kopabiliyor.\nYani şu iki soru arasında henüz otomatik bir uzlaşma döngüsü yok: •\tGerçekte ne çalışıyor? •\tSisteme göre ne çalışıyor olması gerekiyor?\nBugünkü model daha çok “bir kez çalıştır, sonucu kaydet, sonra oraya güven” yaklaşımı. Terraform benzeri bir reconciliation mantığıyla kıyaslayınca, burada geliştirilecek ciddi alan olduğu açık.\nİlginç Teknik Detaylar Her slaydı tekrarlamak istemiyorum (bunun için sunum var), ama birkaç tasarım kararı beni şaşırttığı için bahsetmeye değer:\nIPv4 adreslerini uint32 olarak temsil etmek her şeyi değiştiriyor. 172.19.5.10\u0026lsquo;u tek bir 32-bit tam sayı olarak ifade ettiğinizde, IP havuzu yönetimi aritmetiğe dönüşür: aralık kontrolleri karşılaştırmadır, broadcast hesaplaması bitwise OR\u0026rsquo;dur ve bir alt ağı dolaşmak sadece artırmaktır. Go\u0026rsquo;nun encoding/binary ve net paketleri bunu kolaylaştırır — üçüncü parti IP manipülasyon kütüphanesine gerek yok.\nIP havuzunun çift stratejisi \u0026ldquo;verini tanı\u0026rdquo; ilkesinin güzel bir örneği. Bir /24 alt ağ (254 host) için tüm mevcut IP\u0026rsquo;lerin bir haritasını önceden oluşturmayı göze alabilirsiniz. Bir /8 (16 milyon host) için bu çılgınlık — sadece başlatmak 5 saniye ve ~1 GB bellek alır. Havuz büyük alt ağlarda şeffaf şekilde tembel stratejiye geçer: yalnızca kullanılmış IP\u0026rsquo;leri takip et, rastgele adaylar üret ve dene. Sonuç: /8 başlatma 5,1 saniyeden 0,004 saniyeye düştü.\nCGo kullanmamak kesin bir kısıttı. Ana Go libvirt binding\u0026rsquo;i (libvirt-go) C kütüphanesini sarar, yani libvirt-dev yüklü olması gerekir ve statik binary\u0026rsquo;den vazgeçersiniz. virsh CLI\u0026rsquo;ı os/exec üzerinden kullanmak kuşkusuz daha kırılgan (metin çıktısını parse ediyorsunuz), ama binary\u0026rsquo;yi taşınabilir tutar. WireGuard anahtar üretimi için aynı mantık — golang.org/x/crypto/curve25519, OpenSSL veya wg genkey ihtiyacını ortadan kaldırır.\nGoroutine-ID log\u0026rsquo;lama, bir debug hack\u0026rsquo;iydi ve özelliğe dönüştü. 6 goroutine aynı anda log yazarken, bir korelasyon olmadan neler olduğunu anlamak imkânsız. runtime.Stack çıktısından goroutine ID\u0026rsquo;sini çıkarmak hack\u0026rsquo;çe (bir debug string parse ediyorsunuz), ama işe yarıyor ve paralel yürütmeyi log çıktısında görünür kılıyor. Farklı ID\u0026rsquo;ler + örtüşen zaman damgaları = işlerin sadece eşzamanlı görünmediğinin, gerçekten eşzamanlı olduğunun kanıtı.\nHenüz Keşfetmediğim Fikirler Bunlar daha fazla zamanım olsa yapacağım şeyler:\nBir plan komutu (Terraform gibi). up\u0026lsquo;tan önce tam olarak nelerin oluşturulacağını göster — \u0026ldquo;X ağı oluşturulacak, A, B, C container\u0026rsquo;ları başlatılacak, D VM\u0026rsquo;i tanımlanacak.\u0026rdquo; Kullanıcı inceleyip onaylasın. Özellikle prodüksiyona yakın senaryolarda kullanışlı.\nConfig değişikliğinde canlı yeniden yükleme. YAML dosyasını izle ve çalışan durumu uzlaştır. Config\u0026rsquo;e yeni bir container mi eklendi? Geri kalanı yıkmadan ayağa kaldır. Bir VM mi kaldırıldı? Kapat. Bu özünde bir kontrol döngüsü ve Go\u0026rsquo;nun fsnotify + goroutine\u0026rsquo;leri bunu doğal olarak inşa etmeye uygun.\nİş yükü türleri için plugin sistemi. Şu anda araç tam olarak iki tür iş yükü biliyor: Docker container\u0026rsquo;ları ve libvirt VM\u0026rsquo;leri. Ama kalıp genelleştirilebilir — Podman, Firecracker veya hatta Kata Containers için plugin\u0026rsquo;ler düşünülebilir. Start(), Stop(), Status() metodları olan bir WorkloadDriver arayüzü temiz bir soyutlama olurdu.\nPrometheus metrik endpoint\u0026rsquo;i. Mevcut container sayısı, VM sayısı, IP havuzu kullanım oranı, provisioning gecikmesi bilgilerini dışa aç. Uzun süre çalışan bir lab sunucusu için kapasite planlaması açısından gerçekten yararlı olurdu.\nVM\u0026rsquo;ler için snapshot ve rollback. Bir checkpoint tanımla, öğrencilerin/test edenlerin her şeyi bozmasına izin ver, sonra bilinen iyi duruma geri dön. Libvirt snapshot\u0026rsquo;ları doğal olarak destekler — tesisat basit, sadece bağlanmamış.\ngRPC API. CLI interaktif kullanım için iyi, ama bunu daha büyük bir sisteme (bir eğitim platformuna, bir CI kontrolcüsüne) entegre etmek istiyorsanız, gRPC API uzak istemcilerin ortamları programatik olarak orkestre etmesini sağlar. Cobra CLI ve Orchestrator struct\u0026rsquo;ı zaten temiz bir şekilde ayrılmış durumda, dolayısıyla Up(), Down() ve Status()\u0026lsquo;u gRPC handler\u0026rsquo;larına sarmak mekanik bir iş.\nÇoklu sunucu için WireGuard mesh. Tek yıldız topolojisi yerine, birden fazla orchestrator örneğinin mesh oluşturmasına izin ver. Her biri anahtar üretir, public key\u0026rsquo;lerini ve endpoint\u0026rsquo;lerini değiştirir ve sunucular arasında düz bir ağ elde edersiniz. Eğer bir gün mantıklı olursa, dağıtık versiyona giden yol bu.\nSon Düşünceler Bu araç, Kubernetes\u0026rsquo;in tek sunucu için fazla olması ve shell script\u0026rsquo;lerinin bakımsız kalması yüzünden var. Tatlı nokta, tek bir işi iyi yapan sıkıcı bir Go binary\u0026rsquo;si: tekrarlanabilir bir hibrit ortamı ayağa kaldırmak ve temizce kapatmak.\nGo bu iş için doğru dil çıktı — tek bir özellik yüzünden değil, altyapı araçlarının statik binary, ucuz eşzamanlılık, birinci sınıf ağ desteği ve güçlü standart kütüphane kombinasyonunu istemesi yüzünden. Ekosistemdeki her büyük altyapı projesi (Docker, Kubernetes, Terraform, Prometheus) aynı bahsi yaptı.\nBunlardan herhangi biri iş akışınız için kullanışlı geliyorsa, kod açık ve binary kendi kendine yeterli. Deneyin, kırın ve nelerin eksik olduğunu söyleyin.\n📊 Sunum slaytları · 📊 Slides (English) · 💻 GitHub · LinkedIn\n","permalink":"https://mrturkmen.com/posts/orchestrating-virtual-hub-in-go/","summary":"\u003cp\u003eGeçtiğimiz günlerde \u003cstrong\u003eGophers İstanbul 2026\u003c/strong\u003e\u0026lsquo;da bir sunum yaptım. Konu: tek bir Go binary\u0026rsquo;si ile Docker container\u0026rsquo;ları, libvirt/KVM sanal makineleri, DHCP+DNS altyapısını ve WireGuard VPN tünelini — hepsini bir YAML dosyasından ayağa kaldıran bir araç. Sunumda bol bol kod parçacıkları ve Go\u0026rsquo;nun iç yapısı var. Bu yazı \u003cem\u003eo değil\u003c/em\u003e. Burada daha çok aracın arkasındaki dağınık gerçek-dünya probleminden, nerelerde gerçekten işe yaradığını düşündüğümden, nelerin hâlâ eksik olduğundan ve vakit bulursam denemek istediğim fikirlerden bahsetmek istiyorum.\u003c/p\u003e","title":"Go ile Sanal Ortam Orkestrasyonu"},{"content":"What I like about interviews is that they can directly show your weaknesses to you. When you think about a friend, girlfriend, parents and even your teacher, they all might say that you are great. Well, maybe you are really good at what you do, but hearing it from someone who has objective views on you is very effective. It is all about trade-offs. Here, objective views refer to the perspective of conducting interviews - it would not be fair to have your closest friend as the interviewer. All judgment is done based on your technical skills rather than personal thoughts. Sure, you do not want to hear bad judgments from your close friends, girlfriend or parents. However, you might still want to see your value in the industry independently from what your friends, parents think about. Therefore, I started interviewing with companies to find my weaknesses and desired job.\nI have been rejected by many companies even before the screening, and an automated message appears in your inbox daily. At the beginning, you feel upset, but once you see the general situation, you start to get used to it. I mean, there are some ridiculous job advertisements which request more than six years of experience on certain technologies but are recruiting you as an intern. Despite all these scenarios, it is still not easy to get to the first stage. Nevertheless, I had some full-time position interviews in the past. At least four of them took around 5-6 hours of technical interview with different people each hour. The last one was not an exception, it took seven hours, and I am glad to have this interview regardless of its result.\nI recently had an interview at one of the MAANG companies. I can say that I really liked the process, as all the people in the interview were so nice. Due to the time difference, it was a little bit difficult to arrange interview hours, but it was managed thanks to their understanding and effort. The interview provided me with lots of \u0026ldquo;aha\u0026rdquo; moments, as I understood my existing strengths and weaknesses, even though it was painful. I had a chance to meet and talk with great people, in particular with those who worked on a topic close to my master\u0026rsquo;s thesis topic. After hours of interviews and waiting for the result, I have been rejected for the position. Of course, there might be many reasons behind the rejection, and the first thing that comes to mind is being not good enough. It might be true, even though it might be difficult to accept, but it might not be true as well. The trick, I guess, is to keep continuing to apply and get interviews, then eventually you will see whether it is true or not.\nExcept for the unexpected, I have seen a discussion on quota and I would like to quote an argument from there, I totally agree on this argument.\nThe last interview was composed of all areas in computer science; architecturing, system design, programming, multi-thread programming, however at each step, there was live coding. People are thinking with you on live coding and trying to help you to understand the concept as well as they try to understand how you think. Attitudes of the people were always positive, at least in my experiences. The small details matter when it comes to detailing a concept, programming or understanding the question. Understanding the process of what actually has been asked can be fastened by practicing the questions on leetcode, hackerrank, and other similar websites. Watching videos on Youtube about some concepts is beneficial however they do not really assist you in detailing a concept, at least for me. Technical books, Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, System Design Interview – An insider\u0026rsquo;s guide, Software Architecture: The Hard Parts and many more technical books provide theories behind certain technologies, more details which are asked in the interviews. Therefore, keeping a habit of reading them proactively while practicing on the coding questions websites ( leetcode, hackerrank) will provide all necessary knowledge. I missed some parts due to time constraints that I have at the moment ( -finishing up master thesis, studying, working, etc- ), however I will be back on track after some time. And at the end, a company will try to choose the person who is the best fit for the role :)\nThe outcomes from the failures that I experienced; invest in education, keep trying, work, work, work, no easy solution.\nKeep reading technical blogs and books Practicing coding exercises ( even though for some cases this is not that critical) Review code, check on popular Github repositories Understand the details, focus on small points Try to implement the theory that you have learned during your studies Try to understand huge architectures by dividing the components Reading should be careful, not just checking out, focusing on something which looks easy might make it harder to understand Explain your learnings to someone who has no idea about what you are talking about, if they are just smiling, but no questions asked, then most probably you failed to explain, they are just acting nicely to you :) Apply to the jobs which are relevant to you, do not apply blindly and expect to have returns. When you have a rejection after long hours of interviews, do a post-mortem :) and focus the outcome of the post-mortem, then keep learning, applying and trying Good luck !\n","permalink":"https://mrturkmen.com/posts/rejection/","summary":"\u003cp\u003eWhat I like about interviews is that they can directly show your weaknesses to you. When you think about a friend, girlfriend, parents and even your teacher, they all might say that you are great. Well, maybe you are really good at what you do, but hearing it from someone who has objective views on you is very effective. It is all about trade-offs. Here, objective views refer to the perspective of conducting interviews - it would not be fair to have your closest friend as the interviewer. All judgment is done based on your technical skills rather than personal thoughts. Sure, you do not want to hear bad judgments from your close friends, girlfriend or parents. However, you might still want to see your value in the industry independently from what your friends, parents think about. Therefore, I started interviewing with companies to find my weaknesses and desired job.\u003c/p\u003e","title":"rejection "},{"content":"Have you heard of it ? If not, you are not the only one, who did not heard such a field until now.\nSoftware development should not only focus on development of the software. In those cases, numerous problems will occur eventually, it might resemble addressing the manifestations of a severe ailment without delving into its underlying source.\nThere are some standardizations and certifications which are pointing out main requirements and features of these kind of software systems. I started to learn about this in the aerospace domain, for instance DO-178 safety standard is one of them. Even though DO-178 standard published in 1984, it has been regularly updated in 1985 and 2011.\nIn the book of “A Practical Guide for Aviation Software and 00-178C Compliance”, it is mentioned that there are five key factors which must be thoroughly and constantly addressed while developing safety-critical software.\nWell-documented systems architecture and requirements definition\nSystem requirements should be validated for correctness and completeness. Solid safety practices at all levels of development.\nDisciplined implementation\nWell-defined change management system Requirements and safety attributes must be accurately implemented. No unintended functionality is added. Well-qualified personnel\nCommitment to excellence perfectionists who strive for 100% on every task. Thorough testing at all levels\nBack to the standardization DO-178, it defines five levels A to E. Level E is where no objectives have to be met, and from E to A it increases objectives, eventually Level A has to met all objectives. When DO-178C is checked focus area is mainly lifecycle of processes; software planning, software development, software verification, software quality assurance, software configuration management and certification liaison. In the industry, applying all the steps takes tremendous amount of time at the end, despite of developing and completing it. There is still too much bureaucratic steps to apply in order to have certification for the software which is developed under the standardization.\nThe Table 1 shows, overview of DO-178C processes and sub-processes.\nAs it is mentioned earlier, there are levels from A to E, and they are categorizes with their failure condition category, number of objectives and w. independence.\nSoftware Level Failure Condition Category Objectives w. Independence A Catastrophic 71 30 B Hazardous 69 18 C Major 62 5 D Minor 26 2 E No Safety Effect 0 0 In the end, it is a very challenging and time-consuming process to develop software under DO-178C standardization. However, it is a must for the safety-critical software development in the aerospace domain.\nTo expand objectives and provide more detail, I can provide following table:\nDO-178C Level Objectives Methodology Tools Level A Ensure no software error results in catastrophic failure Formal Methods Model Checking Tools (e.g., SPIN, NuSMV, Alloy) Automated Requirement Traceability Tools Level B Ensure errors don\u0026rsquo;t cause catastrophic failure, but may lead to hazardous/major failures Design Reviews Design Review Tools (e.g., IBM Rhapsody) Configuration Management Tools (e.g., Git) Level C Ensure errors don\u0026rsquo;t cause hazardous conditions, but may lead to major failures Testing Unit Testing Frameworks (e.g., JUnit, Google Test) Code Coverage Tools (e.g., BullseyeCoverage) Level D Ensure errors don\u0026rsquo;t cause major failures, but may lead to minor failures Requirements Management Requirements Management Tools (e.g., IBM DOORS) Formal Methods Tools (e.g., SPARK, CodePeer) Level E Ensure errors don\u0026rsquo;t affect aircraft operation, controllability, or flight safety Design Reviews Model-Based Design Tools (e.g., Simulink) Safety Analysis Tools (e.g., FTA, FMEA software) The journey through the levels of DO-178C certification illuminates the symbiotic relationship between safety and security in airborne software development. There are still many detailed points to consider and work on however to keep the post short, cutting here might be nice for informative part.\nThe aviation industry constitutes a realm of unparalleled complexity, innovation, and global connectivity, embodying a whole new world of exploration and advancement.\nAs baby steps, I started to learn and wanted to put what I read and summarized, here, this might not be interesting information for all however it is just a note to myself.\nOh, before forgetting, military and civil aviation industries have completely different procedures and approaches to develop an aircraft. Indeed, there are some common points where both should obey, for instance flight operations in for military based aircrafts are often conducted in dynamic and potentially hostile environments. In civil aviation industry, flight operations are typically conducted in a more predictable and controlled environment. There are many other points which they differ a lot, but it is obvious.\nReferences:\nA Practical Guide for Aviation Software and DO-178C Compliance, by Leanna Rierson Christoph Torens. \u0026ldquo;Safety Versus Security in Aviation, Comparing DO-178C with Security Standards,\u0026rdquo; AIAA 2020-0242. AIAA Scitech 2020 Forum. January 2020. ","permalink":"https://mrturkmen.com/posts/safety-critical-software/","summary":"\u003cp\u003eHave you heard of it ?  If not, you are not the only one, who did not heard such a field until now.\u003c/p\u003e\n\u003cp\u003eSoftware development should not only focus on development of the software. In those cases, numerous problems will occur eventually, it might resemble addressing the manifestations of a severe ailment without delving into its underlying source.\u003c/p\u003e\n\u003cp\u003eThere are some standardizations and certifications which are pointing out main requirements and features of these kind of software systems. I started to learn about this in the aerospace domain, for instance DO-178 safety standard is one of them. Even though DO-178 standard published in 1984, it has been regularly updated in 1985 and 2011.\u003c/p\u003e","title":"brief intro on DO-178; safety critical software development"},{"content":"Python has some excellent features which provide you confidence. One of them is decorators. Decorators are used to modify the behaviour of a function or a class. In this article, we will learn how to use decorators in Python.\nWhat are decorators? In programming, decorators are functions that enhance the functionality of another function by receiving it as input, modifying it, and then returning a new function. This process is known as metaprogramming, as one part of the program modifies another part during compilation.\nHow to use decorators? In Python, functions are first-class objects. It means that functions can be passed around and used as arguments, just like any other object (string, int, float, list, and so on). Consider the following examples:\n# this function takes another function as an argument, adds functionality and returns another function def uppercase_decorator(func): def wrapper(): result = func() return result.upper() return wrapper @uppercase_decorator # using decorator def say_hello(): return \u0026#34;Hello, world!\u0026#34; print(say_hello()) # Output: HELLO, WORLD! Another example for logging:\ndef logger(func): def wrapper(*args, **kwargs): print(\u0026#39;Logging execution\u0026#39;) func(*args, **kwargs) print(\u0026#39;Done logging\u0026#39;) return wrapper @logger def sample(): print(\u0026#39;-- Inside sample function\u0026#39;) sample() Output of the above code is:\nLogging execution -- Inside sample function Done logging Another example with classes:\nclass LogDecorator: def __init__(self, func): self.func = func def __call__(self, *args, **kwargs): print(f\u0026#34;Calling function: {self.func.__name__}\u0026#34;) return self.func(*args, **kwargs) @LogDecorator def add_numbers(x, y): return x + y result = add_numbers(3, 5) # Output: Calling function: add_numbers print(result) # Output: 8 Another example for authentication:\ndef login_required(func): def wrapper(*args, **kwargs): if is_user_logged_in(): return func(*args, **kwargs) else: raise Exception(\u0026#34;User must be logged in to access this resource.\u0026#34;) return wrapper @login_required def restricted_function(): return \u0026#34;This function requires authentication.\u0026#34; # Usage restricted_function() Another example for processing data with threads and decorators:\nimport concurrent.futures def concurrent_execution(num_threads): def decorator(func): def wrapper(*args, **kwargs): # Create a ThreadPoolExecutor with the specified number of threads with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor: # Submit the function to the executor as a concurrent task future = executor.submit(func, *args, **kwargs) # Wait for the task to complete and retrieve the result result = future.result() return result return wrapper return decorator @concurrent_execution(num_threads=4) def process_data(data): # Perform data processing tasks (e.g., data transformation, aggregation, analysis) # This function is executed concurrently by multiple threads # ... return processed_data # Usage data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] @concurrent_execution(num_threads=4) def process_data(data): # Perform data processing tasks (e.g., data transformation, aggregation, analysis) # This function is executed concurrently by multiple threads # ... return processed_data # Usage data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] @concurrent_execution(num_threads=4) def process_data(data): # Perform data processing tasks (e.g., data transformation, aggregation, analysis) # This function is executed concurrently by multiple threads # ... return processed_data # Usage data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] # Decorated function will be executed concurrently by four threads result = process_data(data) print(result) References Python Decorators-Documentation Datacamp Decorators ChatGPT :') ","permalink":"https://mrturkmen.com/posts/decorators-in-python/","summary":"\u003cp\u003ePython has some excellent features which provide you confidence. One of them is decorators. Decorators are used to modify the behaviour of a function or a class. In this article, we will learn how to use decorators in Python.\u003c/p\u003e\n\u003ch2 id=\"what-are-decorators\"\u003eWhat are decorators?\u003c/h2\u003e\n\u003cp\u003eIn programming, decorators are functions that enhance the functionality of another function by receiving it as input, modifying it, and then returning a new function. This process is known as metaprogramming, as one part of the program modifies another part during compilation.\u003c/p\u003e","title":"decorators in python with examples"},{"content":"It was another interview that I had with a company, and I was asked about metaclasses in Python. Up to now, I did not have a requirement to use them, it leads to have a superficial idea about them but not exactly know what they are, therefore decided to do some research and write a short blog post about them.\nWhat are metaclasses? In Python, everything is an object. Classes are objects too. Metaclasses are classes that create classes or customize class creation. We can say that metaclasses are the \u0026ldquo;stuff\u0026rdquo; that creates classes.\nYou define classes in order to create objects, right? But we learned that Python classes are objects. Well, metaclasses are what create these objects. They are the classes\u0026rsquo; classes, you can picture them this way:\nclass MetaLogger(type): def __new__(cls, name, bases, attrs): # Add logging functionality to all methods in the class for attr_name, attr_value in attrs.items(): if callable(attr_value): # Wrap the method with logging attrs[attr_name] = cls.add_logging(attr_value) return super().__new__(cls, name, bases, attrs) @staticmethod def add_logging(func): def wrapper(*args, **kwargs): print(f\u0026#34;Calling method: {func.__name__}\u0026#34;) return func(*args, **kwargs) return wrapper class MyClass(metaclass=MetaLogger): def foo(self): print(\u0026#34;Executing foo method.\u0026#34;) def bar(self): print(\u0026#34;Executing bar method.\u0026#34;) # Create an instance of MyClass obj = MyClass() # Call methods on the instance obj.foo() obj.bar() In the given piece of code, we have a metaclass called MetaLogger that wraps all methods of the class with logging functionality. We can see that we have a class called MyClass that uses MetaLogger as a metaclass. When we create an instance of MyClass, we can see that all methods are wrapped with logging functionality. For example, when we call obj.foo(), we can see that the method is wrapped with logging functionality.\nImagine that you define a generic logger metaclass that adds logging functionality to all methods of the class. You can use this metaclass in all classes that you want to add logging functionality to all methods. This is only one example of the power of metaclasses.\nThe output of the above code is:\nCalling method: foo Executing foo method. Calling method: bar Executing bar method. Create classes on the fly Metaclasses can be used to create classes on the fly. For example, you can create a class on the fly that has a method that returns the current time. Here is an example:\nimport datetime class TimeGetter(type): def __new__(cls, name, bases, attrs): attrs[\u0026#39;get_current_time\u0026#39;] = lambda self: datetime.datetime.now() return super().__new__(cls, name, bases, attrs) # Create a class dynamically using the TimeGetter metaclass DynamicClass = TimeGetter(\u0026#39;DynamicClass\u0026#39;, (), {}) # Create an instance of the dynamically created class obj = DynamicClass() # Call the method to get the current time current_time = obj.get_current_time() # Print the current time print(\u0026#34;Current Time:\u0026#34;, current_time) In the given piece of code, we have a metaclass called TimeGetter that adds a method called get_current_time to the class. We can see that we create a class called DynamicClass dynamically using the TimeGetter metaclass. When we create an instance of DynamicClass, we can see that we have a method called get_current_time that returns the current time.\nThe output of the above code is ( time will be different of course):\nCurrent Time: 2023-06-18 12:20:00.000000 The power of metaclasses is astonishing. You can create classes on the fly, you can add functionality to all methods of a class, you can do many things with metaclasses. Although metaclasses are powerful, they are not used very often. You can use metaclasses to create following features with a super effective way;\nDSL (Domain Specific Language) a framework a plugin system a database ORM (Object Relational Mapper) serialization/deserialization system a caching system a validation system These are only a few examples of what you can do with metaclasses. Try to use them in your projects, you will see that they are very powerful.\nConclusion Python is like an ocean and you can learn new things every day. Metaclasses are one of the most powerful features of Python. I have tried to give general information and few examples of metaclasses. However, there are many great, extensive and well written articles about metaclasses. I recommend you to read them. First and foremost, I recommend you to read the official documentation. As a second or this could also be first step as well, I strongly recommend you to read the following articles to get more depth knowledge about metaclasses:\nWhat are metaclasses in Python? Metaclasses in 500 lines of code I hope, you learnt something new today.\nReferences Official Documentation What are metaclasses in Python? Metaclasses in 500 lines of code And ChatGPT of course :) ","permalink":"https://mrturkmen.com/posts/metaclasses-python/","summary":"\u003cp\u003eIt was another interview that I had with a company, and I was asked about metaclasses in Python. Up to now, I did not have a requirement to use them, it leads to have a superficial idea about them but not exactly know what they are, therefore decided to do some research and write a short blog post about them.\u003c/p\u003e\n\u003ch2 id=\"what-are-metaclasses\"\u003eWhat are metaclasses?\u003c/h2\u003e\n\u003cp\u003eIn Python, everything is an object. Classes are objects too. Metaclasses are classes that create classes or customize class creation. We can say that metaclasses are the \u0026ldquo;stuff\u0026rdquo; that creates classes.\u003c/p\u003e","title":"from interview question to enlightenment: metaclasses in python "},{"content":"It is always been fun to access a local machine that is running at your home or at university or somewhere which does not have a public IP address or you do not have control over the network.\nI already wrote a post about cloudflare tunneling however it was about serving local web applications. This time I will show you how to access local SSH service without exposing it to the world.\nI was not thinking to write a post about this, however, before knowing cloudflare tunneling, I was considering renting a static IP address from my ISP. However, it costs some money, and also from a security point of view, it is not a good idea to expose your local machine to the world. ( somehow we do already but that\u0026rsquo;s a different topic :) )\nPrerequisites A domain managed by Cloudflare (free tier is enough) You can check out how to add your domain to Cloudflare here Steps to be followed on the remote machine Once you have a domain managed by Cloudflare, you can start to configure your remote machine. You need to install cloudflared cli. (The machine that you want to access remotely through SSH).\nInstallation and configuring clouflared tool is very well explained in Cloudlfare\u0026rsquo;s documentation.\nPlease follow the steps of configuring cloudlfared tool from the official documentation.\nWhen you are done with logging in and configuring cloudflared tool. You can create a tunnel to your local machine and configure it with SSH service.\n# create a tunnel $ cloudflared tunnel create my-tunnel You will get a tunnel ID. You will need this ID to configure your local machine.\nNow you need to create a route to the tunnel. (The domain is the domain that you have added to Cloudflare.)\n# create a route $ cloudflared tunnel route dns my-tunnel remote.mydomain.com The SSH service or any other service can be configured from CLI however it will be ephemeral.\nTo make it persistent, you need to create a configuration file under ~/.cloudflared/config.yml.\ntunnel: my-tunnel credentials-file: /home/awesomeuser/.cloudflared/credentials-file.json url: ssh://localhost:22 You can find the full documentation https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/local/local-management/configuration-file/.\nOnce you created the configuration file (config.yml), then you can execute the install command to serve your local SSH service through the Cloudflare tunnel.\n$ cloudflared --config ~/.cloudflared/config.yml tunnel service install Apart from tunneling, since we are exposing SSH service, it is a rule of thumb to use SSH keys instead of a password.\n# change PasswordAuthentication value in /etc/ssh/sshd_config file and restart SSH service $ sudo sed -i \u0026#39;s/PasswordAuthentication yes/PasswordAuthentication no/g\u0026#39; /etc/ssh/sshd_config $ sudo service ssh restart These are the steps that you need to follow on the machine that you would like to access remotely through SSH.\nSteps to be followed on the client machine Once you have configured your remote machine, you need to install cloudflared cli on your client machine too. (The machine which you use to access the remote machine through SSH)\nYou can follow the same installation instructions from https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/installation.\nOn the client side, you need to configure SSH for the host which uses cloudflare tunnel.\nAdd the following lines to the file ~/.ssh/config.\nHost myhost HostName remote.mydomain.com IdentityFile ~/.ssh/id_rsa ProxyCommand cloudflared access ssh --hostname %h User awesomeuser I assumed that you already shared keys with your remote machine.\nIf you did not, you can follow the steps from https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-on-ubuntu-1804.\nAdditionally, please follow best practices for SSH configuration from https://www.ibm.com/support/pages/best-practices-ssh-configuration).\nNow you can access your remote machine through SSH.\n$ ssh myhost That\u0026rsquo;s it. You can access your remote machine from anywhere in the world. No need to have a public IP address or move your machine with you when traveling.\n","permalink":"https://mrturkmen.com/posts/tunnelling-to-local-ssh-service/","summary":"\u003cp\u003eIt is always been fun to access a local machine that is running at your home or at university or somewhere which does not have a public IP address or you do not have control over the network.\u003c/p\u003e\n\u003cp\u003eI already wrote a post about \u003ca href=\"https://mrturkmen.com/posts/cloudflare-tunneling/\"\u003ecloudflare tunneling\u003c/a\u003e however it was about serving local web applications. This time I will show you how to access local SSH service without exposing it to the world.\u003c/p\u003e","title":"cloudflare tunneling: serve local SSH service without PUBLIC IP"},{"content":"In this post, I will create a simple command to run a workflow on Github using AWS Gateway service. This post was written when AWS offered 1 million free API calls per month for a 12-month period.\nSetup Github The setup on the Github side is minimal and only requires adding a line to the existing workflow file. For demonstration purposes, I will go use one of the repository from merkez. Let\u0026rsquo;s choose insthat repository for this occurence.\nWhen we check its existing workflow file, it looks like as follows:\nname: Test installation script on: push: paths: - \u0026#39;install-tools.sh\u0026#39; schedule: # 15:05 UTC \u0026gt; 17:05 CEST - cron: \u0026#39;5 15 * * *\u0026#39; workflow_dispatch: jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Test all functions run: | sudo chmod +x ./install-tools.sh sudo bash install-tools.sh --random The workflow file for the insthat repository is likely one of the simplest you can find. In order to trigger the workflow using an API call, we will need to add repository_dispatch after workflow_dispatch. For more information on repository_dispatch, please check out the following link: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#repository_dispatch\nHere are the lines that should be added after workflow_dispatch:\nrepository_dispatch: types: on-demand-run The final workflow file for the insthat repository can be found at this link: https://github.com/merkez/insthat/blob/main/.github/workflows/test-script.yml\nIn order to make an API call to trigger the workflow, you will need to generate a Personal Access Token (PAT) with the necessary permissions, at a minimum the \u0026ldquo;Repo\u0026rdquo; option needs to be selected. More information on how to generate a PAT can be found here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token\nThese are the only steps required on the Github side, no other setup is necessary.\nSetup AWS Gateway API To set up the API Gateway on AWS, you will need to log in to the AWS Console, navigate to the API Gateway service, and select the \u0026ldquo;Build REST API\u0026rdquo; option from the list of options available. Once you are in the \u0026ldquo;Build REST API\u0026rdquo; page, select the \u0026ldquo;Rest API\u0026rdquo; and \u0026ldquo;New API\u0026rdquo; options as shown in the screenshot.\nFill out API name and description (optional) and click Create.\nIn opened window, click \u0026ldquo;Create Resource\u0026rdquo; and fill out input fields according to your preference and create it.\nUnder the \u0026ldquo;Resources\u0026rdquo; section, you will need to create a new POST method by clicking on the \u0026ldquo;Actions\u0026rdquo; button. This will open a new window, in this window you need to select HTTP as Integration Type. To call the workflow on Github, you need to construct the link. The template for the Github link that we will use is https://api.github.com/repos/{owner}/{repository}/dispatches. In this demonstration, the link will be https://api.github.com/repos/merkez/insthat/dispatches after inserting the specific values.\nWhen we applied save, we should be able to see following page:\nAfter setting the HTTP link, you will need to navigate to the \u0026ldquo;Integration Request\u0026rdquo; box in order to set up the authentication keys and request body for the Github API. To authenticate the request, the following headers need to be set:\nKey Value Accept \u0026lsquo;application/vnd.github.v3+json\u0026rsquo; Authorization \u0026lsquo;Bearer {PERSONAL_ACCESS_TOKEN}\u0026rsquo; Adjust \u0026ldquo;Mapping Templates\u0026rdquo; section with Slack\u0026rsquo;s content type, which is application/x-www-form-urlencoded.\nThe body, {\u0026quot;event_type\u0026quot;:\u0026quot;on-demand-run\u0026quot;} should match what you had in repository_dispath types.\nLastly, deploy API on AWS Gateway. https://t76xrsn8z6.execute-api.us-east-1.amazonaws.com/execute-workflow\nSet stage input fields.\nAfter deployment, navigate to Stages, and check full \u0026ldquo;Invoke URL\u0026rdquo; as shown below:\nFor our case Invoke URL is: https://t76xrsn8z6.execute-api.us-east-1.amazonaws.com/execute-workflow/runner-api\nThis URL will be used when creating the slack command.\nWe can now setup Slack command and give a try through Slack.\nSetup Slack App To create a Slash command, you need to create a Slack application on the workspace that you have permissions to. For the purpose of this demonstration, I will use the https://mrkzi.slack.com workspace.\nOnce you have created the Slack application, you will need to go to the \u0026ldquo;Slash Commands\u0026rdquo; option in the application page of Slack and click on \u0026ldquo;Create New Command\u0026rdquo;. Slack will ask for some information, you can choose any command name you want to use. The most important field in this section is the Request URL, which should be retrieved from the AWS Gateway (Invoke URL) as explained in the previous step when setting up the AWS Gateway.\nAfter you have completed the steps above, you will need to install the app you have created to the workspace by going to the \u0026ldquo;Basic Information\u0026rdquo; section of the app.\nWhen it is done, you can go to Slack desktop application or on web, open workspace, you should be able to see autobot-runner or whatever you call it under apps of the workspace.\nThen, type the command you generated, for this demonstration it is, /run-insthat, when it is typed following option will appear. Execute it and let Github execute the workflow :)\nOnce the command /run-insthat is executed on Slack, workflow will automatically run.\nWith this setup, you will be able to trigger the workflow on Github through Slack from any device that has the Slack app, such as a phone, tablet, or PC. This flexibility increases the level of convenience and ease of use, making it a more efficient and effective way to manage workflows.\n(from https://xkcd.com/2268)\n","permalink":"https://mrturkmen.com/posts/automate-ci-cd-with-slack-command/","summary":"\u003cp\u003eIn this post, I will create a simple command to run a workflow on Github using AWS Gateway service. This post was written when AWS offered 1 million free API calls per month for a 12-month period.\u003c/p\u003e\n\u003ch2 id=\"setup-github\"\u003eSetup Github\u003c/h2\u003e\n\u003cp\u003eThe setup on the Github side is minimal and only requires adding a line to the existing workflow file. For demonstration purposes, I will go use one of the repository from \u003ca href=\"https://github.com/merkez\"\u003emerkez\u003c/a\u003e. Let\u0026rsquo;s choose \u003ca href=\"https://github.com/merkez/insthat\"\u003einsthat\u003c/a\u003e repository for this occurence.\u003c/p\u003e","title":"automate: run github ci/cd through slack slash command"},{"content":"Intro git cherry-pick is a powerful subcommand of git extensive information can be found here: https://www.atlassian.com/git/tutorials/cherry-pick\nIn this mini blog post, I will demonstrate how I just used it to remove specific commits from git history and preserve rest of the information as it is. (e.g metadata, timestamps, author, committer information)\ncherry-pick is generally used to remove some of the commits from git history, or re-order commit history. In this example, I will go through simple use case of it with combination of git filter-branch.\nThe story I created a private repository months ago, and consistenly filled it up with solutions of some exercises for a course at the university. However, I pushed official slide documents which I should not, luckily the repository was private and no one has an access to it. In time, I kept pushing more stuff to the repo, then we came to end of the semester. I thought that, it is better to make it public without leaking official solutions. Therefore, I had to remove the commits which are included earlier to the git history. The screenshot indicates which commits needs to be removed. It is indeed easy process to complete if it is done correctly.\nSteps to complete Need to create a new branch before the document is committed to main/master branch\non main/master:\ngit checkout -b new-main All commits except highlighted ones needs to be cheery-picked.\ncherry-pick commits, except highlighted ones :\ngit cherry-pick \u0026lt;commit-id\u0026gt; cherry-pick command will pick the commits that you select and append them to head of new branch. However, the problem is that it will NOT preserve timestamp information. It means that even though you made some commits months ago or a week ago, it will be replaced with the time that you cheerry-picked the commits. This behaviour breaks git history of a repository unexpectedly. The ideal scenario would be to have same information ( -timestamp-) as before. Therefore, there is one more additional step.\nSet original timestamp information on new branch for all commits.\nset git committer date to author date:\ngit filter-branch --env-filter \u0026#39;export GIT_COMMITTER_DATE=\u0026#34;$GIT_AUTHOR_DATE\u0026#34;\u0026#39; Finally, you can push new-main branch to an upstream.\nEnd result \u0026ndash;\u0026gt; https://github.com/mrtrkmn/cloud-computing/commits/main\n","permalink":"https://mrturkmen.com/posts/cherry-pick/","summary":"\u003ch2 id=\"intro\"\u003eIntro\u003c/h2\u003e\n\u003cp\u003e\u003ccode\u003egit cherry-pick\u003c/code\u003e is a powerful subcommand of \u003ccode\u003egit\u003c/code\u003e extensive information can be found here: \u003ca href=\"https://www.atlassian.com/git/tutorials/cherry-pick\"\u003ehttps://www.atlassian.com/git/tutorials/cherry-pick\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eIn this mini blog post, I will demonstrate how I just used it to remove specific commits from git history and preserve rest of the information as it is. (e.g metadata, timestamps, author, committer information)\u003c/p\u003e\n\u003cp\u003echerry-pick is generally used to remove some of the commits from git history, or re-order commit history. In this example, I will go through simple use case of it with combination of \u003ccode\u003egit filter-branch\u003c/code\u003e.\u003c/p\u003e","title":"cherry-pick: re-build git history"},{"content":"Gitbook is simple, easy and free way of keeping notes under your own domain. However, it is NOT possible to set password protection when you do not want anyone access to your resources except you or your team for free. (More information: https://docs.gitbook.com/features/visitor-authentication) Nice news is that it is possible to set password protection even though you are on free tier with some tricks.\nFirst of all, for this example, your custom domain should be configured to Cloudflare, since I will use Cloudflare Workers to setup Basic Authentication in between you and Gitbook origin.\nI assumed that your custom domain is already configured with Cloudflare, if not, configure it, if you want to use Cloudflare Workers feature.\nCloudflare worker has some request limitation in free tier (Accounts using the Workers Free plan are subject to a daily request limit of 100,000 requests.). However, it is not a problem for us since we won\u0026rsquo;t run e-commerce website or something. It is just private notes which is accesible from anywhere when you have the credentials.\nHere is an example code block for Basic Authentication\n(Taken and modified from: https://developers.cloudflare.com/workers/examples/basic-auth/ )\n/** * Shows how to restrict access using the HTTP Basic schema. * @see https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication * @see https://tools.ietf.org/html/rfc7617 * * A user-id containing a colon (\u0026#34;:\u0026#34;) character is invalid, as the * first colon in a user-pass string separates user and password. */ const BASIC_USER = \u0026#39;randomusername\u0026#39;; const BASIC_PASS = \u0026#39;randompass\u0026#39;; addEventListener(\u0026#39;fetch\u0026#39;, event =\u0026gt; { event.respondWith( handleRequest(event.request).catch(err =\u0026gt; { const message = err.reason || err.stack || \u0026#39;Unknown Error\u0026#39;; return new Response(message, { status: err.status || 500, statusText: err.statusText || null, headers: { \u0026#39;Content-Type\u0026#39;: \u0026#39;text/plain;charset=UTF-8\u0026#39;, // Disables caching by default. \u0026#39;Cache-Control\u0026#39;: \u0026#39;no-store\u0026#39;, // Returns the \u0026#34;Content-Length\u0026#34; header for HTTP HEAD requests. \u0026#39;Content-Length\u0026#39;: message.length, }, }); }) ); }); /** * Receives a HTTP request and replies with a response. * @param {Request} request * @returns {Promise\u0026lt;Response\u0026gt;} */ async function handleRequest(request) { const { protocol, pathname } = new URL(request.url); // In the case of a Basic authentication, the exchange // MUST happen over an HTTPS (TLS) connection to be secure. if (\u0026#39;https:\u0026#39; !== protocol || \u0026#39;https\u0026#39; !== request.headers.get(\u0026#39;x-forwarded-proto\u0026#39;)) { throw new BadRequestException(\u0026#39;Please use a HTTPS connection.\u0026#39;); } switch (pathname) { // case \u0026#39;/\u0026#39;: // return new Response(\u0026#39;Anyone can access the homepage.\u0026#39;); // case \u0026#39;/logout\u0026#39;: // // Invalidate the \u0026#34;Authorization\u0026#34; header by returning a HTTP 401. // // We do not send a \u0026#34;WWW-Authenticate\u0026#34; header, as this would trigger // // a popup in the browser, immediately asking for credentials again. // return new Response(\u0026#39;Logged out.\u0026#39;, { status: 401 }); default: { // The \u0026#34;Authorization\u0026#34; header is sent when authenticated. if (request.headers.has(\u0026#39;Authorization\u0026#39;)) { // Throws exception when authorization fails. const { user, pass } = basicAuthentication(request); if (verifyCredentials(user, pass)) { return await fetch(request) } } // Not authenticated. return new Response(\u0026#39;You need to login.\u0026#39;, { status: 401, headers: { // Prompts the user for credentials. \u0026#39;WWW-Authenticate\u0026#39;: \u0026#39;Basic realm=\u0026#34;Private Area\u0026#34;, charset=\u0026#34;UTF-8\u0026#34;\u0026#39;, }, }); } case \u0026#39;/favicon.ico\u0026#39;: case \u0026#39;/robots.txt\u0026#39;: return new Response(null, { status: 204 }); } return new Response(\u0026#39;Not Found.\u0026#39;, { status: 404 }); } function verifyCredentials(user, pass) { if (BASIC_USER !== user || BASIC_PASS !== pass) { return false } return true } /** * Parse HTTP Basic Authorization value. * @param {Request} request * @throws {BadRequestException} * @returns {{ user: string, pass: string }} */ function basicAuthentication(request) { const Authorization = request.headers.get(\u0026#39;Authorization\u0026#39;); const [scheme, encoded] = Authorization.split(\u0026#39; \u0026#39;); // The Authorization header must start with Basic, followed by a space. if (!encoded || scheme !== \u0026#39;Basic\u0026#39;) { throw new BadRequestException(\u0026#39;Malformed authorization header.\u0026#39;); } // Decodes the base64 value and performs unicode normalization. // @see https://datatracker.ietf.org/doc/html/rfc7613#section-3.3.2 (and #section-4.2.2) // @see https://dev.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String/normalize const buffer = Uint8Array.from(atob(encoded), character =\u0026gt; character.charCodeAt(0)); const decoded = new TextDecoder().decode(buffer).normalize(); // The username \u0026amp; password are split by the first colon. //=\u0026gt; example: \u0026#34;username:password\u0026#34; const index = decoded.indexOf(\u0026#39;:\u0026#39;); // The user \u0026amp; password are split by the first colon and MUST NOT contain control characters. // @see https://tools.ietf.org/html/rfc5234#appendix-B.1 (=\u0026gt; \u0026#34;CTL = %x00-1F / %x7F\u0026#34;) if (index === -1 || /[\\0-\\x1F\\x7F]/.test(decoded)) { throw new BadRequestException(\u0026#39;Invalid authorization value.\u0026#39;); } return { user: decoded.substring(0, index), pass: decoded.substring(index + 1), }; } function UnauthorizedException(reason) { this.status = 401; this.statusText = \u0026#39;Unauthorized\u0026#39;; this.reason = reason; } function BadRequestException(reason) { this.status = 400; this.statusText = \u0026#39;Bad Request\u0026#39;; this.reason = reason; } This is the domain to be protected:\nhttps://worker.mrturkmen.com\nThis subdomain contains an example Gitbook, which is synced with Github repo.\nhttps://github.com/merkez/worker-gitbook\nYou can follow the steps given below from web dashboard of Cloudflare:\nCreate worker service Step 1: Create worker service\nCreate trigger Step 2: Create trigger, add route to custom domain created for Gitbook\nEdit Service Step 3: Edit Service, click QUICK EDIT button\nModify Worker Code Step 4: Paste Authentication code to editor in left side and deploy\nVisit website The website is only accesible with provided username and password in the example code above.\nCloudflare workers has a lot of capabilites not limited to this simple approach. More examples and information can be found in provided links below:\nhttps://workers.cloudflare.com https://developers.cloudflare.com/workers/examples/ https://developers.cloudflare.com/workers/examples/basic-auth/ ","permalink":"https://mrturkmen.com/posts/cloudflare-workers/","summary":"\u003cp\u003eGitbook is simple, easy and free way of keeping notes under your own domain.\nHowever, it is NOT possible to set password protection when you do not want anyone access to your resources except you or your team for free. (\u003cem\u003eMore information: \u003ca href=\"https://docs.gitbook.com/features/visitor-authentication\"\u003ehttps://docs.gitbook.com/features/visitor-authentication\u003c/a\u003e\u003c/em\u003e) Nice news is that it is possible to set password protection even though you are on free tier with some tricks.\u003c/p\u003e\n\u003cp\u003eFirst of all, for this example, your custom domain should be configured to Cloudflare, since I will use Cloudflare Workers to setup Basic Authentication in between you and Gitbook origin.\u003c/p\u003e","title":"cloudflare workers: add auth to free Gitbook space"},{"content":"Would like to see it in action right away ? Go to demonstration video here:\nI am always amazed with Cloudflare\u0026rsquo;s products, blog posts and tools that they are offering. In my spare time, I am trying to read through blog posts over here Cloudflare Blog. Some of them are really easy to consume, some are not, you might need to have an idea about network related terms, or need to follow Cloudflare closely to catch on the blog posts.\nRecently, when I was surfing on Cloudflare website, I came crossed with a tool that I have seen for the first time, which is called cloudflared\nIt is a client of Argo Tunnel which is another amazing product that Cloudflare provide, including for free tiers. I will not describe or explain it in detail, since Cloudflare Blog and documentation here: Argo Tunnel explains it very well. Furthermore, over there, you can see and experience other cool products as well.\nInstead, in this blog post, I am going to use cloudflared tool, to serve a website which is locally running on my PC to the world thanks to Cloudflare Tunnel. Since its guidelines and instructions are crystal clear, some steps might be seen as repetitive of what Cloudflare explained. Neverthless, I would like to do it in an unofficial style anyway.\nI assume that you have hosted your custom domain at Cloudflare. Otherwise, do it as described here: Register a new domain. After installing the tool as described on official page of Cloudflared tool, you can authenticate it with the domain that you will work on. Once authentication is done, following steps can be followed.\nHere is some commands which can easily be grasped at first glance.\n# complete steps and download certificates to authenticate user and domain to use $ cloudflared tunnel login # create a tunnel with name mrtrkmn-tunnel $ cloudflared tunnel create mrtrkmn-tunnel # serve locally running web application on local.mrturkmen.com globally ## this step is not required when you already have ## a config file as described below $ cloudflared tunnel --hostname local.mrturkmen.com --url http://localhost:8000 # create a route to the tunnel over local.mrturkmen.com # it creates a CNAME record on Cloudflare. $ cloudflared tunnel route dns mrtrkmn-tunnel local.mrturkmen.com # run the tunnel $ cloudflared tunnel run Cloudflared tool has a configuration option under your home directory, as shown here:\nI setup localhost:8000 to be served through mrtrkmn-tunnel (ea6cf5bf-13b4-41c5-9000-14705384d83a).\nDifferent websites can be served through different tunnels by specifying newly created tunnel IDs and corresponding URLs. For demonstration purposes, I will only consider one example.\nTotal number of commands after authentication of cloudflared tool is five including running your local deployment.\nCreate tunnel $ cloudflared create tunnel mrtrkmn-tunnel Tunnel credentials written to \u0026lt;your-home-dir\u0026gt;/.cloudflared/961486d1-c624-46b3-8eb5-f74ba8ab2a91.json. cloudflared chose this file based on where your origin certificate was found. Keep this file secret. To revoke these credentials, delete the tunnel. Create tunnel config url: \u0026#34;http://localhost:8000\u0026#34; tunnel: mrtrkmn-tunnel credentials-file: \u0026lt;your-home-dir\u0026gt;/.cloudflared/ea6cf5bf-13b4-41c5-9000-14705384d83a.json. Route tunnel traffic to a subdomain $ cloudflared tunnel route dns mrtrkmn-tunnel local.mrturkmen.com Run HTTP server or a website on given port in config file Just for proof of concept, run a dummy HTTP server to serve globally.\n$ python3 -m http.server --bind 0.0.0.0 Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ... Run tunnel $ cloudflared tunnel run mrtrkmn-tunnel Now, you should be able to see, your local deployment or HTTP(s) server is available on given subdomain in routing step above.\nThere are other cool stuff that Cloudflare is providing even though you have free tier account. Since I am quite interested on Cloudflare tech stack and its products, I will try to test out their products and create simple blog posts. It is now much easy to test out your local deployments with your teammates, friends and other communities.\nReferences Cloudflare Tunnel, https://www.cloudflare.com/products/tunnel/ Cloudflare Zero Trust - Get Started, https://developers.cloudflare.com/cloudflare-one/setup/ Cloudflare Zero Trust - Add web applications, https://developers.cloudflare.com/cloudflare-one/applications/configure-apps/ ","permalink":"https://mrturkmen.com/posts/cloudflare-tunneling/","summary":"\u003ch3 id=\"would-like-to-see-it-in-action-right-away-\"\u003eWould like to see it in action right away ?\u003c/h3\u003e\n\u003cp\u003eGo to demonstration video here:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://www.youtube.com/watch?v=Ayjk7J_KQ8A\"\u003e\u003cimg alt=\"Demonstration Video\" loading=\"lazy\" src=\"https://img.youtube.com/vi/Ayjk7J_KQ8A/0.jpg\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003eI am always amazed with Cloudflare\u0026rsquo;s products, blog posts and tools that they are offering. In my spare time, I am trying to read through blog posts over here \u003ca href=\"https://blog.cloudflare.com/\"\u003eCloudflare Blog\u003c/a\u003e. Some of them are really easy to consume, some are not, you might need to have an idea about network related terms, or need to follow Cloudflare closely to catch on the blog posts.\u003c/p\u003e","title":"cloudflare tunneling: serve local like not local"},{"content":" I am alive !!!\nRight, I did not post anything for long time and could not imagine that this will be my next blog post after long time, but it is indeed.\nThis is a story about a company who hold the product (- sent for warranty -) for five months and did not respond any emails.\nLet\u0026rsquo;s start from the beginning of the story, the product itself is not important, instead how they (- company -) approached to situation is important.\nI have sent the product to Karaca (- krc.com.tr -) in November 2021 by hoping that it will be fixed and returned in two or three weeks at most. However, it turns out that it will not be the case, just realized after two or three weeks later :) At that time, I did not give any importance to the situation instead waited more, more and more.\nAt the end of four months, I decided to take an action, I have tried to reach them through their \u0026ldquo;customer services\u0026rdquo; phone number ( +90 850 2525 572 ). However it connects you to a stupid automated replies, do you remember those ? it replies like, e.g \u0026ldquo;if you would like to learn about your product, press 1 \u0026ldquo;, \u0026ldquo;if you would like to learn about warranty process, press 2\u0026rdquo; and so on. Although I tried all possibilities to reach them through phone, I could not be successfull enough to get them on the line.\nAfterwards, I realized that they are actually responding to questions/complains if you contact with them through their contact form. ( - https://www.krc.com.tr/contact-form - ) At the beginning, I sent some messages through this form, however they replied me with same message everytime.\nThe message was:\nYou have an active record with the number \u0026ldquo;XXX-XXXXXXX-XXXXXX\u0026rdquo; when the necessary control is made based on your request. You will receive a response as soon as possible.\nIt was really annoying to receive same replies without providing any information about why they did not sent back the product to me all these past four months. Since I annoyed to them, I decided to do an automated way of sending messages ( - by providing same message content - ) through their contact form using Python and Github actions.\nI can not deal with them everyday, but automation can :)\nSince there will not be explanation of the code, you can think that it is just a piece of code which automatically fills form and send it using bs4 and selenium.\n( - The code may include some bugs or unneceassary statements, you may want to update it to re-use)\nYou can see it in action from readme file of the project.\nGithub workflow file contains cron job and as well as manually execution of the code.\non: workflow_dispatch: # enable manual run inputs: git-ref: description: Git Ref (Optional) required: false schedule: # cron job run each day at 10.00 AM according to GMT+03:00 - cron: \u0026#39;0 7 * * *\u0026#39; The repository is using secrets to fill out the contact form. Everyday, it sends message to krc.com.tr, takes screenshot of contact page and commits it to screenshots folder and finally finishes the process.\n- name: Complain on KRC run: | python main.py env: PHONE_NUMBER: ${{ secrets.PHONE_NUMBER }} COMPLAIN_MESSAGE: ${{ secrets.COMPLAIN_MESSAGE }} EMAIL: ${{ secrets.EMAIL }} Push changes (- screenshot -) with timestamp using a bot user to the repository.\n- name: Commit SS push run: | git config --global user.email \u0026#34;robotcuk@randommail.com\u0026#34; git config --global user.name \u0026#34;robotcuk\u0026#34; git add \u0026#39;screenshots/contact-page-*.png\u0026#39; git commit -m \u0026#34;${{ steps.date.outputs.date }} Complain is done to KRC on ${{ steps.date.outputs.date }}\u0026#34; git push origin -f main env: GITHUB_TOKEN: ${{ secrets.ROBOTCUK }} All code regarding to described process can be found here: https://github.com/merkez/krccomplain\nTo see it in action you can check out its readme file or watch it on youtube: https://youtu.be/zXrbjEpA_20\nIt can be used for anyone else who would like to complain to https://www.krc.com.tr through contact form until they update the website or put reCAPTCHA version 2.\nOh by the way, the rest of the story and updates about it, is given on project readme file, check it out if you wonder how it is ended up. ( spoiler alert: - not as fantastic as you may think - )\nRight, I like to automate the stuff.\n","permalink":"https://mrturkmen.com/posts/automation-for-complain/","summary":"\u003chr\u003e\n\u003cblockquote\u003e\n\u003cp\u003eI am alive !!!\u003c/p\u003e\u003c/blockquote\u003e\n\u003cblockquote\u003e\n\u003cp\u003eRight, I did not post anything for long time and  could not imagine that this will be my next blog post after long time, but it is indeed.\u003c/p\u003e\u003c/blockquote\u003e\n\u003chr\u003e\n\u003cp\u003eThis is a story about a company who hold the product (- sent for warranty -) for five months and did not respond any emails.\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s start from the beginning of the story, the product itself is not important, instead how they (- company -) approached to situation is important.\u003c/p\u003e","title":"github-actions: complain messages to a company in automated way"},{"content":"Imagine a scenario where you have a monolithic application which uses a config file to store information about log directories, cert dirs and other service information. As an example to it following config file (- it is taken and modified from Haaukins project which I work on- ) can be considered:\nhost: http: myapplication.mrturkmen.com port: insecure: 8080 secure: 8081 tls: enabled: false certfile: \u0026#34;/home/mrturkmen/certs/cert.crt\u0026#34; certkey: \u0026#34;/home/mrturkmen/certs/cert.key\u0026#34; cafile: \u0026#34;/home/mrturkmen/certs/ca.crt\u0026#34; files: ova-directory: \u0026#34;/home/mrturkmen/ova\u0026#34; users-file: \u0026#34;/home/mrturkmen/configs/users.yml\u0026#34; exercises-file: \u0026#34;/home/mrturkmen/configs/exercises.yml\u0026#34; frontends-file: \u0026#34;/home/mrturkmen/configs/frontends.yml\u0026#34; prodmode: true vpn-service: grpc: \u0026#34;vpnservice.mrturkmen.com:4000\u0026#34; auth-key: random-auth-key sign-key: random-sign-key tls: enabled: true certfile: \u0026#34;/home/mrturkmen/certs/cert.crt\u0026#34; certkey: \u0026#34;/home/mrturkmen/certs/cert.key\u0026#34; cafile: \u0026#34;/home/mrturkmen/certs/ca.crt\u0026#34; In this config file we have some set of keys which are defined to be used inside the application, however let\u0026rsquo;s say we would like to update some values from the config file. Then in normal cases (-if no hot reload kind of function implemented- ), user needs to restart entire application. It means application will have some down time, it may be less or more however it is not good way of doing it, in particular to update only a value from config file.\nIn this point, os signals can be used to update config file without restarting or closing the application. There are some other libraries which it is possible to enable watch on config file mode. It means the library will immediately notify entire application when there is change on the config file. However in this scenario, I assume that there is no such a library or framework is integrated.\nHere I am considering the situation from Go language perspective, this may differ or not needed for some programming languages or frameworks.\nChannel and go routine will be used to listen any SIGHUP signal to the process of the application.\nFirst of all, it is nice to create the function which will re-assign config variable of the application when SIGHUP signal received.\nfunc (a *application) ReloadConfig(confFile *string) error { conf, err := NewConfigFromFile(*confFile) if err != nil { return err } a.conf = conf // re-assign applicaiton config file return nil } When SIGNUP signal received by user given function above needs to be called to update configuration file.\nHere is the code which listens any SIGHUP signal to the application\nfunc handleHotConfigReload(confFile *string, reload func(confFile *string) error) { c := make(chan os.Signal, 1) // channel to wait os.Signal signal.Notify(c, syscall.SIGHUP) go func() { // go routine to do not block other requests on the application \u0026lt;-c log.Info().Msgf(\u0026#34;Hot reload for config file...\u0026#34;) if err := reload(confFile); err != nil { log.Error().Msgf(\u0026#34;Error on reloading config file: %s\u0026#34;, err) os.Exit(1) } log.Info().Msgf(\u0026#34;Config is updated !\u0026#34;) }() } This function can be called before or after the application started. It can be called as shown below:\nhandleHotConfigReload(confFilePtr, func(confFile *string) error { return a.ReloadConfig(confFilePtr) // a is application struct }) Then it can be tested with :\n$ kill -SIGHUP \u0026lt;process-id\u0026gt; It will create SIGHUP signal on the process to call ReloadConfig function.\nThis is how OS Signal can be used to update configuration file in an application which is written in Go, when you do not have already implemented library or framework.\n","permalink":"https://mrturkmen.com/posts/hot-reload-with-os-signals/","summary":"\u003cp\u003eImagine a scenario where you have a monolithic application which uses a config file to store information about log directories, cert dirs and other service information. As an example to it following config file (- it is taken and modified from Haaukins project which I work on- ) can be considered:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003ehost\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003ehttp\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003emyapplication.mrturkmen.com\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eport\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003einsecure\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e8080\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003esecure\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003e8081\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003etls\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003eenabled\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003ecertfile\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/certs/cert.crt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003ecertkey\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/certs/cert.key\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003ecafile\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/certs/ca.crt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efiles\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003eova-directory\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/ova\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003eusers-file\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/configs/users.yml\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003eexercises-file\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/configs/exercises.yml\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003efrontends-file\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/configs/frontends.yml\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eprodmode\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003evpn-service\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003egrpc\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;vpnservice.mrturkmen.com:4000\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003eauth-key\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003erandom-auth-key\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003esign-key\u003c/span\u003e: \u003cspan style=\"color:#ae81ff\"\u003erandom-sign-key\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e  \u003cspan style=\"color:#f92672\"\u003etls\u003c/span\u003e:\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003eenabled\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003ecertfile\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/certs/cert.crt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003ecertkey\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/certs/cert.key\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003ecafile\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;/home/mrturkmen/certs/ca.crt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIn this config file we have some set of keys which are defined to be used inside the application, however let\u0026rsquo;s say we would like to update some values from the config file. Then in normal cases (-if no hot reload kind of function implemented- ), user needs to restart  entire application. It means application will have some down time, it may be less or more however it is not good way of doing it, in particular to update only a value from config file.\u003c/p\u003e","title":"go[channels]: hot config reload with os signal "},{"content":" THE REPOSITORY: https://github.com/merkez/ubuntu-packer\nIn this blog post, provisioning and customizing images using packer will be shown with a template repository.\nIf you are asking or wondering what is Packer, the official definition is :\nPacker is a free and open source tool for creating golden images for multiple platforms from a single source configuration. (From Official Website).\nThis post includes provisioning of ubuntu image on AWS and local.\nBuild Custom Ubuntu 20.04 LTS on Local In an ideal repository of Packer template, it would be nice to have a skeleton where it includes uploads, http, scripts folders along packer configuration file with a readme. Overall, the structure of folder might look like this :\n├── http │ └── preseed.cfg # required to change defualt values of ubuntu image ├── readme.md # readme file to have instructions about what to do ├── scripts # scripts/ dir, includes scripts to run on custom image │ ├── cleanup.sh # cleans up /tmp │ ├── install_tools.sh # installs custom tools │ └── setup.sh # setting up config in system wise ├── ubuntu-20.04.json # packer config for ubuntu 20.04 └── uploads # directory to upload files to custom image └── .gitkeep In this setup, http/preseed.cfg defines answers to the questions which may be asked during installation of Ubuntu operating system. More information regarding to preseed.cfg file can be checked from its wiki\nscripts folder composed of bash scripts, chef, ansible or any other installer configuration files or scripts which will install customized tools and define settings of ubuntu image.\nuploads folder includes all files, deb packages, or any other files which will be copied to image which will be inside customized image.\nAnatomy of Packer Configuration File Any packer file composed of three main components which are ;\nBuilders Define the desired platform and platform configurations, including API Key information and desired source images. Example snippet is given from the Packer file:\n\u0026#34;builders\u0026#34;: [ { \u0026#34;boot_command\u0026#34;: [ \u0026#34;\u0026lt;esc\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;\u0026lt;esc\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;/install/vmlinuz\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; auto\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/ask_detect=false\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/layoutcode=us\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/modelcode=pc105\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; debconf/frontend=noninteractive\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; debian-installer=en_US\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; fb=false\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; initrd=/install/initrd.gz\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; kbd-chooser/method=us\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; keyboard-configuration/layout=USA\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; keyboard-configuration/variant=USA\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; locale=en_US\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; netcfg/get_domain=vm\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; netcfg/get_hostname=ubuntu\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; grub-installer/bootdev=/dev/sda\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; noapic\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; -- \u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34; ], \u0026#34;boot_wait\u0026#34;: \u0026#34;10s\u0026#34;, \u0026#34;format\u0026#34;: \u0026#34;ova\u0026#34;, \u0026#34;disk_size\u0026#34;: 25240, \u0026#34;guest_additions_path\u0026#34;: \u0026#34;VBoxGuestAdditions_{{.Version}}.iso\u0026#34;, \u0026#34;guest_os_type\u0026#34;: \u0026#34;Ubuntu_64\u0026#34;, \u0026#34;headless\u0026#34;: true, \u0026#34;http_directory\u0026#34;: \u0026#34;http\u0026#34;, \u0026#34;iso_checksum\u0026#34;: \u0026#34;sha256:f11bda2f2caed8f420802b59f382c25160b114ccc665dbac9c5046e7fceaced2\u0026#34;, \u0026#34;iso_urls\u0026#34;: [ \u0026#34;iso/ubuntu-20.04.1-legacy-server-amd64.iso\u0026#34;, \u0026#34;https://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.04/release/ubuntu-20.04.1-legacy-server-amd64.iso\u0026#34; ], \u0026#34;shutdown_command\u0026#34;: \u0026#34;echo \u0026#39;ubuntu\u0026#39;|sudo -S shutdown -P now\u0026#34;, \u0026#34;ssh_password\u0026#34;: \u0026#34;ubuntu\u0026#34;, \u0026#34;ssh_port\u0026#34;: 22, \u0026#34;ssh_timeout\u0026#34;: \u0026#34;10000s\u0026#34;, \u0026#34;ssh_username\u0026#34;: \u0026#34;ubuntu\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;virtualbox-iso\u0026#34;, \u0026#34;vboxmanage\u0026#34;: [ [ \u0026#34;modifyvm\u0026#34;, \u0026#34;{{.Name}}\u0026#34;, \u0026#34;--memory\u0026#34;, \u0026#34;2048\u0026#34; ], [ \u0026#34;modifyvm\u0026#34;, \u0026#34;{{.Name}}\u0026#34;, \u0026#34;--cpus\u0026#34;, \u0026#34;1\u0026#34; ] ], \u0026#34;virtualbox_version_file\u0026#34;: \u0026#34;.vbox_version\u0026#34;, \u0026#34;vm_name\u0026#34;: \u0026#34;ubuntu_vm_ubuntu_20_{{timestamp}}\u0026#34; } ] In the builders config, we are defining some set of keys in JSON file, which are very obvious from its name, we are considering to build image locally. All the keys are important in given builders config however most important and might need to update time to time is iso_urls which are the places where packer download iamges and customize it according to your scripts. Another crucial key is to have headless value true which means that there will be no GUI running when packer command is executed to run the Packer JSON file.\nProvisioner Defines how to configure the image most likely by your using existing configuration management tools like Ansible, Chef, Puppet or pure bash scripts.\nIn our example, bash scripts will be provided to install tools and update configuration of ubuntu image to make it customized. Provisioner section of a Packer JSON file can be seen as below:\n\u0026#34;provisioners\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;:\u0026#34;uploads\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;/home/ubuntu\u0026#34; }, { \u0026#34;execute_command\u0026#34;: \u0026#34;echo \u0026#39;ubuntu\u0026#39; | {{.Vars}} sudo -S -E bash \u0026#39;{{.Path}}\u0026#39;\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;scripts/install_tools.sh\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34; }, { \u0026#34;execute_command\u0026#34;: \u0026#34;echo \u0026#39;ubuntu\u0026#39; | {{.Vars}} sudo -S -E bash \u0026#39;{{.Path}}\u0026#39;\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;scripts/setup.sh\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34; }, { \u0026#34;execute_command\u0026#34;: \u0026#34;echo \u0026#39;ubuntu\u0026#39; | {{.Vars}} sudo -S -E bash \u0026#39;{{.Path}}\u0026#39;\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;scripts/cleanup.sh\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34; } ] Here we are defining existing bash scripts in order to execute in the process of customizing Ubuntu image. The steps under provisioners are pretty clear.\nThe content of uploads file will be uploaded to home directory /home/ubuntu\nIn second step, install_tools.sh will be executed and other steps will be followed in order.\nPost Processors Related to the builder, runs after the image is built, it is generally used to generate or apply artifacts. In this example, it is not required however more information can be found here: post processors\nCommunicator How packer works on the machine image during the creation. By default it is over SSH communication and it does not need to be defined explicitly. More information can be found here: communicator\nOver all packer file can be seen as follow:\n{ \u0026#34;builders\u0026#34;: [ { \u0026#34;boot_command\u0026#34;: [ \u0026#34;\u0026lt;esc\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;\u0026lt;esc\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;/install/vmlinuz\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; auto\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/ask_detect=false\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/layoutcode=us\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/modelcode=pc105\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; debconf/frontend=noninteractive\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; debian-installer=en_US\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; fb=false\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; initrd=/install/initrd.gz\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; kbd-chooser/method=us\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; keyboard-configuration/layout=USA\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; keyboard-configuration/variant=USA\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; locale=en_US\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; netcfg/get_domain=vm\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; netcfg/get_hostname=ubuntu\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; grub-installer/bootdev=/dev/sda\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; noapic\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; -- \u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34; ], \u0026#34;boot_wait\u0026#34;: \u0026#34;10s\u0026#34;, \u0026#34;format\u0026#34;: \u0026#34;ova\u0026#34;, \u0026#34;disk_size\u0026#34;: 25240, \u0026#34;guest_additions_path\u0026#34;: \u0026#34;VBoxGuestAdditions_{{.Version}}.iso\u0026#34;, \u0026#34;guest_os_type\u0026#34;: \u0026#34;Ubuntu_64\u0026#34;, \u0026#34;headless\u0026#34;: true, \u0026#34;http_directory\u0026#34;: \u0026#34;http\u0026#34;, \u0026#34;iso_checksum\u0026#34;: \u0026#34;sha256:f11bda2f2caed8f420802b59f382c25160b114ccc665dbac9c5046e7fceaced2\u0026#34;, \u0026#34;iso_urls\u0026#34;: [ \u0026#34;iso/ubuntu-20.04.1-legacy-server-amd64.iso\u0026#34;, \u0026#34;https://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.04/release/ubuntu-20.04.1-legacy-server-amd64.iso\u0026#34; ], \u0026#34;shutdown_command\u0026#34;: \u0026#34;echo \u0026#39;ubuntu\u0026#39;|sudo -S shutdown -P now\u0026#34;, \u0026#34;ssh_password\u0026#34;: \u0026#34;ubuntu\u0026#34;, \u0026#34;ssh_port\u0026#34;: 22, \u0026#34;ssh_timeout\u0026#34;: \u0026#34;10000s\u0026#34;, \u0026#34;ssh_username\u0026#34;: \u0026#34;ubuntu\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;virtualbox-iso\u0026#34;, \u0026#34;vboxmanage\u0026#34;: [ [ \u0026#34;modifyvm\u0026#34;, \u0026#34;{{.Name}}\u0026#34;, \u0026#34;--memory\u0026#34;, \u0026#34;2048\u0026#34; ], [ \u0026#34;modifyvm\u0026#34;, \u0026#34;{{.Name}}\u0026#34;, \u0026#34;--cpus\u0026#34;, \u0026#34;1\u0026#34; ] ], \u0026#34;virtualbox_version_file\u0026#34;: \u0026#34;.vbox_version\u0026#34;, \u0026#34;vm_name\u0026#34;: \u0026#34;ubuntu_vm_ubuntu_20_{{timestamp}}\u0026#34; } ], \u0026#34;provisioners\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;:\u0026#34;uploads\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;/home/ubuntu\u0026#34; }, { \u0026#34;execute_command\u0026#34;: \u0026#34;echo \u0026#39;ubuntu\u0026#39; | {{.Vars}} sudo -S -E bash \u0026#39;{{.Path}}\u0026#39;\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;scripts/install_tools.sh\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34; }, { \u0026#34;execute_command\u0026#34;: \u0026#34;echo \u0026#39;ubuntu\u0026#39; | {{.Vars}} sudo -S -E bash \u0026#39;{{.Path}}\u0026#39;\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;scripts/setup.sh\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34; }, { \u0026#34;execute_command\u0026#34;: \u0026#34;echo \u0026#39;ubuntu\u0026#39; | {{.Vars}} sudo -S -E bash \u0026#39;{{.Path}}\u0026#39;\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;scripts/cleanup.sh\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34; } ], \u0026#34;variables\u0026#34;: { \u0026#34;version\u0026#34;: \u0026#34;0.1\u0026#34; } } How to run locally This file can be run from the place where ubuntu-20.04.json file is located.\n$ packer build ubuntu-20.04.json It will start to build custom image by installing tools which are defined under scripts and configure username and password according to preseed.cfg and setup.sh files.\nBuild Custom Ubuntu 20.04 LTS on Cloud It is more practical and preferrable to use if you already have an cloud option to consider. This packer configuration will create custom image directly on cloud and save it to AMIs to your AWS account.\nThe anatomy of packer files is similar, only section which needs to be changed compared to local one, is builders section. It is defining all required AWS variables and AMIs to customize.\nAs an cloud example AWS will be used to create custom image.\nBuilders on Cloud \u0026#34;builders\u0026#34;: [ { \u0026#34;type\u0026#34;:\u0026#34;amazon-ebs\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;{{user `aws_region`}}\u0026#34;, \u0026#34;access_key\u0026#34;: \u0026#34;{{user `aws_access_key`}}\u0026#34;, \u0026#34;secret_key\u0026#34;: \u0026#34;{{user `aws_secret_key`}}\u0026#34;, \u0026#34;subnet_id\u0026#34;: \u0026#34;{{user `aws_subnet_id`}}\u0026#34;, \u0026#34;security_group_id\u0026#34;: \u0026#34;{{user `aws_security_group`}}\u0026#34;, \u0026#34;source_ami_filter\u0026#34;: { \u0026#34;filters\u0026#34;: { \u0026#34;virtualization-type\u0026#34;: \u0026#34;hvm\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;ubuntu/images/*ubuntu-focal-20.04-amd64-server-*\u0026#34;, \u0026#34;root-device-type\u0026#34;: \u0026#34;ebs\u0026#34; }, \u0026#34;owners\u0026#34;: [\u0026#34;099720109477\u0026#34;], \u0026#34;most_recent\u0026#34;: true }, \u0026#34;instance_type\u0026#34;: \u0026#34;{{user `instance_type`}}\u0026#34;, \u0026#34;ssh_username\u0026#34;:\u0026#34;ubuntu\u0026#34;, \u0026#34;ami_name\u0026#34;: \u0026#34;ubuntu-ami-custom_{{timestamp}}\u0026#34; } ] In this configuration, all keys are important to consider, however there are some which are crucial and required to run it. More information about the keys can be found here: Amazon AMI Builder\nWe would like to create a custom Ubuntu-20.04 image on cloud and save it as AMI to run it later, we are searching its pattern from available AMIs on AWS Management Console or it can be found through out this website : https://cloud-images.ubuntu.com/locator/ec2/\nOnce you have declared which AMI to customize, it needs to be located under source_ami_filter with wildcards and owners. Setting most_recent to true means that when this Packer JSON file is executed it will fetch and customize last updated AMI.\nAccess Key, Secret Key are required and should not be exposed to public in any moment, if exposed, they need to be updated immediately. They will be used to communicate with AWS to fire up instances to create custom image according to given settings defined in builders and provisioners.\nThe values of keys are defined in variables and parsed from out of it.\n\u0026#34;variables\u0026#34;: { \u0026#34;aws_access_key\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;aws_secret_key\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;aws_region\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;aws_vpc\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;aws_subnet\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;ami_name\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;ami_description\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;builder_name\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;username\u0026#34;:\u0026#34;ubuntu\u0026#34;, \u0026#34;instance_type\u0026#34;:\u0026#34;t2.medium\u0026#34;, \u0026#34;tarball\u0026#34;: \u0026#34;\u0026#34; }, In variables section, username, instance_type, aws_access_key, aws_secret_key variables should be set correctly to create the image on cloud. Other variables are optional and variables section can be populated more.\nCustomize settins on Cloud On cloud builds, cloud configuration file should be used instead of preseed.cfg to customize settings. The defaults.cfg file where it contains custom settings such as default username, password, changing visudo file and more. Example defaults.cfg can be as follow:\n#cloud-config system_info: default_user: name: ubuntu sudo: [\u0026#34;ALL=(ALL) NOPASSWD:ALL\u0026#34;] lock_passwd: false plain_text_passwd: \u0026#39;ubuntu\u0026#39; More information regarding to defaults.cfg file can be found here and customized more: https://cloudinit.readthedocs.io/en/latest/topics/examples.html\nHow to run Once variables are set, it can be run in same way with the local one.\n$ packer build aws_packer.json Complete packer JSON file : aws_packer.json\nAs a summary, Packer is really cool tool to use to automate the process of creating custom images and it can be used for Dockers as well. For local example in this post, it will produce OVA file to import, on cloud it will generate custom AMI under your AWS account.\nAll scripts and config files can be found in this repository: https://github.com/merkez/ubuntu-packer\n","permalink":"https://mrturkmen.com/posts/build-with-packer/","summary":"\u003chr\u003e\n\u003cp\u003e\u003cstrong\u003eTHE REPOSITORY: \u003ca href=\"https://github.com/merkez/ubuntu-packer\"\u003ehttps://github.com/merkez/ubuntu-packer\u003c/a\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIn this blog post, provisioning and customizing images using packer will be shown with a template repository.\u003c/p\u003e\n\u003cp\u003eIf you are asking or wondering what is Packer, the official definition is :\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003ePacker is a free and open source tool for creating golden images for multiple platforms from a single source configuration.  (From Official Website).\u003c/p\u003e\u003c/blockquote\u003e\n\u003cp\u003eThis post includes provisioning of ubuntu image on AWS and local.\u003c/p\u003e\n\u003ch1 id=\"build-custom-ubuntu-2004-lts-on-local\"\u003eBuild Custom Ubuntu 20.04 LTS on Local\u003c/h1\u003e\n\u003cp\u003eIn an ideal repository of Packer template, it would be nice to have a skeleton where it includes \u003ccode\u003euploads\u003c/code\u003e, \u003ccode\u003ehttp\u003c/code\u003e, \u003ccode\u003escripts\u003c/code\u003e folders along packer configuration file with a readme. Overall, the structure of folder might look like this :\u003c/p\u003e","title":"packer: build custom images on cloud and local "},{"content":"fail2ban A while ago, I was checking servers\u0026rsquo; logs to see any suspicious activities going on from outside. I noticed that the servers both staging/testing and production servers are receiving a lot of brute force SSH attacks from variety of countries which are shown in table below.\nList of IP Addresses ( who are doing SSH Brute Forcing ) ** Information on the table gathered from: [ https://www.maxmind.com/en/geoip-demo ]\nBan failed attempts Although servers have no password login, they are kept brute forcing on SSH port. Well, fail2ban was one of obvious solution to block those IP addresses permanently or temporarily. I prefered to block them all permanently until manual unblocking has been done by me.\nThe steps for installing fail2ban is pretty obvious, you are doing same things like, apt-get update \u0026amp;\u0026amp; apt-get install fail2ban. After installation completed, configuration is much more important.\nFollowing steps will guide you to block any ip address who are brute forcing on SSH.\nCopy template file $ cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local Set Ban time\nIt is possible to set ban time permanent or temporarily. I preffered to setup permanent, so for this reason I have changed bantime = -1. Save and exit from the file when you are done.\n$ vim /etc/fail2ban/jail.conf # Permanent ban bantime = -1 Create custom rules for SSH $ vim /etc/fail2ban/jail.d/sshd.local [sshd] enabled = true port = ssh filter = sshd logpath = /var/log/auth.log # place of ssh logs maxretry = 4 # maximum number of attempts that user can do (*Maxretry value and log file can be changed according to your setup.)\nMake the rules persistent\nIn order to make the rules persistent which means, the blocked IPs will not be deleted after restart of fail2ban service or restart of server. It requires to have some tricks to be done inside iptables rules under fail2ban. Add following cat and echo commands at the end of actionstart and actionban respectively .\n$ vim /etc/fail2ban/action.d/iptables-multiport.conf . . . actionstart = iptables -N fail2ban-\u0026lt;name\u0026gt; iptables -A fail2ban-\u0026lt;name\u0026gt; -j RETURN iptables -I \u0026lt;chain\u0026gt; -p \u0026lt;protocol\u0026gt; -m multiport --dports \u0026lt;port\u0026gt; -j fail2ban-\u0026lt;name\u0026gt; cat /etc/fail2ban/persistent.bans | awk \u0026#39;/^fail2ban-\u0026lt;name\u0026gt;/ {print $2}\u0026#39; \\ | while read IP; do iptables -I fail2ban-\u0026lt;name\u0026gt; 1 -s $IP -j \u0026lt;blocktype\u0026gt;; done . . . actionban = iptables -I fail2ban-\u0026lt;name\u0026gt; 1 -s \u0026lt;ip\u0026gt; -j \u0026lt;blocktype\u0026gt; echo \u0026#34;fail2ban-\u0026lt;name\u0026gt; \u0026lt;ip\u0026gt;\u0026#34; \u0026gt;\u0026gt; /etc/fail2ban/persistent.bans Save and restart service $ systemctl restart fail2ban These are most basic steps to block IP addresses who are actively brute forcing to servers. After some time, I am able to see them with following command :)\n$ sudo fail2ban-client status sshd Status for the jail: sshd |- Filter | |- Currently failed:\t12 | |- Total failed:\t107 | `- File list:\t/var/log/auth.log `- Actions |- Currently banned:\t16 |- Total banned:\t16 `- Banned IP list:\t171.239.254.84 184.102.70.222 180.251.85.85 103.249.240.208 159.65.194.150 117.217.35.114 113.164.79.129 61.14.228.170 116.110.30.245 43.239.80.181 77.222.130.223 14.255.137.219 184.22.195.230 125.25.82.12 116.110.109.90 115.76.168.231 It is growing in time however at least they are not able to brute force the server with same IP addresses. There are plenty of other ways to make SSH port much more secure and effective however I think having updated ssh daemon/client, passwordless login and fail2ban will be enough in most of the cases. Therefore, while I was doing this stuff, although there are plenty of guides over there, I wanted to note down how I did it to come back and check if something happens.\nTake care !\n","permalink":"https://mrturkmen.com/posts/fail2ban/","summary":"\u003ch1 id=\"fail2ban\"\u003efail2ban\u003c/h1\u003e\n\u003cp\u003eA while ago, I was checking servers\u0026rsquo; logs to see any suspicious activities going on from outside. I noticed that the servers both staging/testing and production servers are receiving a lot of brute force SSH attacks from variety of countries which are shown in table below.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"list-of-ip-addresses--who-are-doing-ssh-brute-forcing-\"\u003eList of IP Addresses ( who are doing SSH Brute Forcing )\u003c/h2\u003e\n\u003c!-- raw HTML omitted --\u003e\n\u003cp\u003e** Information on the table gathered from: [ https://www.maxmind.com/en/geoip-demo ]\u003c/p\u003e","title":"fail2ban: block ssh bruteforce attacks "},{"content":"In this post, deployment process of an application with Ansible will be explained. Traditionally applications can be deployed in different ways, quite similar approach to deploy applications like in Ansible is executing bash script which has ssh commands. To give an example, Travis continuous integration has a feature where a bash script can be defined to deploy application and through given instructions within bash script, application can be successfully deployed.\nDetails regarding to deployment using Travis bash scripting can be found here\nTravis Script Deployment I would like to give an real case example from one of the project which I work on. We were using Travis script deployment for a while and it works pretty well. The bash script which I use in our deployment process is given below:\n#!/usr/bin/env bash f=dist/hknd_linux_amd64/hknd amigo=./svcs/amigo user=ntpd hostname=sec02.lab.es.aau.dk keyfile=./travis_deploy_key deploy_path=/data/home/ntpd/daemon/hknd amigo_path=/data/home/ntpd/daemon/svcs/amigo if [ -f $f ]; then echo \u0026#34;Deploying \u0026#39;$f\u0026#39; to \u0026#39;$hostname\u0026#39;\u0026#34; chmod 600 $keyfile ssh -i $keyfile -o StrictHostKeyChecking=no $user@$hostname sudo /bin/systemctl stop hknd.service scp -i $keyfile -o StrictHostKeyChecking=no $f $user@$hostname:$deploy_path scp -i $keyfile -r -o StrictHostKeyChecking=no $amigo $user@$hostname:$amigo_path ssh -i $keyfile -o StrictHostKeyChecking=no $user@$hostname sudo /bin/systemctl start hknd.service else echo \u0026#34;Error: $f does not exist\u0026#34; exit 1 fi As you can observe from the bash script, every step of the deployment is given as ssh/scp commands. There is no harm regarding to it as long as it contains few steps. However, as time pass more configurations, applications will required to be deployed, updated, modified and checked, then it might turn into headache. Therefore, having well structured deployment steps using Ansible will put us to safe side.\nBefore jumping into deployment with Ansible, I would like to point out some factors which can be counted as disadvantages of not integrating Ansible to deployment process.\nNot common way of utilizing resources Not well structured deployment scripts which has high potential of being not working very well. Having plain ssh commands increase likelihood of having issues regarding to settings, deployments and more. There are many more drawbacks of using pure bash scripts in deployment process, however, these issues may not be applicable for all them.\nFor our case, I would like to convert our bash script given above to Ansible which has more elegant structure and easy to manage.\nMove to Ansible Since the bash script does not contain complex instructions, it would be very easy to convert it into Ansible playbooks. Before starting to convert it into Ansible playbook, necessary ssh connection should be set correctly for development and production environments. (- test environment as well if required -).\nSetting ssh connection between server and ansible user is pretty straitforward, it contains following steps;\nGenerate SSH Key pair Copy public key to authorized_keys on server side Encrypt private key Have decrypt script to use private key on CI without compromising it. Overall simplified flow for deployment is given below :\nAs it is declared from overall picture above, we need to provide encrypted ssh key and script for decryption together, in order to use plain private key to access the server.\nIn this setup, Github CI will be control node which will have access to server where we would like to deploy the application.\nLet\u0026rsquo;s start to complete steps,\nGenerate SSH Key pair\n$ ssh-keygen You can keep everything default or provide some information about the questions when you run it. Once, execution of command finished, there will be public and private key, you need to append the public key to user\u0026rsquo; authorized keys file on server. Afterwards, connection should be established, you may want to test it using traditional ssh command.\nEncrypt Private Key\nIn order to use the ssh key which is generated before, we need to encrypt the key, I preferred to use gpg tool, there are many examples about it on internet, you can check it if you wish.\n$ gpg --symmetric --cipher-algo AES256 \u0026lt;private-key-file\u0026gt; The command will prompt you to provide passphrase to encrpyt and decrypt the private key when required. Choose strong and long passphrase. Once it is done, include encrpyted file into git. (- which means commit it as well-)\nOnce they have completed, the rest is structing Ansible playbook to deploy the file to server.\nExample Repo I am going to create a repository on Github to demonstrate what I have described earlier in action.\nFor the demo purposes, I will upload a service file to server and start it, simplified version of given bash commands above.\nAnsible playbook will contain following;\nStopping already running service Changing binary file of the service Starting it again The tasks can be extended according to user needs however to keep it short and show how Ansible could be used on continuous integration, I will continue to have minimal playbook.\nLink to example repository: https://github.com/merkez/ansible-deploy\nThe structure of the repository as following:\nAs it can be observed from the figurre above, I have only three tasks which are combined under main.yml.\nSome configuration regarding to Ansible, such as private key, inventory file location declaration is saved to file ansible.cfg among ssh connection configuration.\nInventory file contains server(s) to deploy the application.\nThis post is not about how to write ansible playbooks, hence, I am going to skip to explain it. If you would like to check and understand it you can check following repositories for examples;\nDevOps Learning Journey Handwritten notes about Ansible Decrypt script is crucial file which is decrypting encrypted private key to access the server.\nDO NOT FORGET TO SET YOUR SECRET_PASSPHRASE TO SECRETS OF THE REPOSITORY\nThe workflow file The workflow file for this repository is pretty straitforward to create as well, what needs to be done is that ansible should be installed into environment. Afterwards, running ansible playbook command after decrpyting the encrypted private key will complete the tasks.\nThe generated workflow is for giving demonstration, in normal production case, the pipeline should NOT be broken, each step from testing to production deployment should be as much as automated.\nThe completed workflow file:\n# This is a basic workflow to help you get started with Actions name: CI # Controls when the action will run. on: # Triggers the workflow on tagged commits push: tags: - \u0026#39;*.*.*\u0026#39; # Allows you to run this workflow manually from the Actions tab workflow_dispatch: # A workflow run is made up of one or more jobs that can run sequentially or in parallel jobs: # This workflow contains a single job called \u0026#34;build\u0026#34; build: # The type of runner that the job will run on runs-on: ubuntu-latest # Steps represent a sequence of tasks that will be executed as part of the job steps: # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it - uses: actions/checkout@v2 - name: Install Ansible run: | sudo apt update -y sudo apt install software-properties-common -y sudo apt-add-repository --yes --update ppa:ansible/ansible sudo apt install ansible -y - name: Set Execute command to bash script run: chmod +x ./.github/scripts/decrypt.sh # Runs a single command using the runners shell - name: Decrypt large secret run: ./.github/scripts/decrypt.sh env: SECRET_PASSPHRASE: ${{ secrets.SECRET_PASSPHRASE }} - name: Escalate Private Key Permissions run: chmod 400 ~/.privkey - name: Run ansible command run: | ansible-playbook -i ./inventory main.yml env: ANSIBLE_CONFIG: ./ansible.cfg - name: Clean Key run: rm -rf ~/.privkey The final result from Github actions:\nKeep in mind that this is just a minor portion of a long pipeline which has all unit tests, checks, linting and integration tests. Without proper pipeline in place, having Ansible might not be logical or required. Consider your cases when you would like to move to deployment with Ansible.\nCheers !\n","permalink":"https://mrturkmen.com/posts/deploy-with-ansible/","summary":"\u003cp\u003eIn this post, deployment process of an application with Ansible will be explained. Traditionally applications can be deployed in different ways, quite similar approach to deploy applications like in Ansible is executing bash script which has ssh commands. To give an example, Travis continuous integration has a feature where a bash script can be defined to deploy application and through given instructions within bash script, application can be successfully deployed.\u003c/p\u003e","title":"ansible: deploy easily in simple steps"},{"content":"While watching video tutorial about Ansible, I took some notes and created following PDF file.\nINTRODUCTION TO ANSIBLE HANDWRITTEN NOTES\n","permalink":"https://mrturkmen.com/posts/introduction-to-ansible-notes/","summary":"\u003cp\u003eWhile watching video tutorial about Ansible, I took some notes and created following PDF file.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"../../pdfs/20201205-introduction-to-ansible.pdf\"\u003eINTRODUCTION TO ANSIBLE HANDWRITTEN NOTES\u003c/a\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"../../images/ansible_hand_written_notes.png\"\u003e\u003c/p\u003e","title":"ansible: introductory handwritten notes"},{"content":"In some moments, Youtube algorithm is working perfect, but sometimes it shows a video from ten years ago from nowhere. For the moments where it shows and suggests videos/playlists to us, we might want to save the list of playlist and watch in some other time. It could be on a plane, train, bus, whenever you are planning to spent some time. However, taking the URL of a playlist and saving it to your cute note program might not be sufficient enough. There is a high chance that it will be forgotten or missed, therefore, I thought that it would be nice to have an automated way of saving playlists on somewhere and download them when I need. (- in particular, when there is no or limited internet connection -).\nIn this blog post, I will go through a simple project which downloads all the videos in a playlist and generates seperate tar.gz files for each playlist to release on Github using Github actions.\nRequirements Whenever starting a project, it is always nice to imply divide and conquer approach if you know what you would like to achieve. Divide and conquer approach will hugely assist you during the development and planning no matter what is the size of the project. As first step, let\u0026rsquo;s define what we need for accomblishing such a thing. ]\nA library/program which downloads Youtube videos from given URL. A library/program which compress the downloaded videos to minimize the size. A workflow on Github Actions to trigger releases. When the given components are clarified, only one more step left to have. There should be main component which combines the requirements given above. For this purpose, I will use Go programming language.\nAvailable Tools When existing libraries, tools and open source projects checked for the first requirement, there are some on Github, namely;\nannie: 👾 Fast, simple and clean video downloader youtube-dl: Command-line program to download videos from YouTube.com and other video sites you-get: Dumb downloader that scrapes the web ytdl: YouTube download library and CLI written in Go These are the tools which enable users to download Youtube videos by providing the URL or the ID.(- some of them supports different social media platforms too, e.g vimeo -).\nTo make things simpler, I chose to use youtube-dl, since it is more promising than others and formats the output very well according to user output pattern.\nAnother point is to clarify which tool/library should I use to compress the downloaded videos from Youtube. With help of a little bit googling, I found out that pigz is quite nice tool which compress given folder/file in paralel by using all available cores of the machine. The second requirement is cleared as well, now it is time to combine both of them in one and add Github workflow on top it.\nI will mention about the Github workflow file, after structure of the program which automates the process.\nCode Structure To make things faster (- in terms of development time -), I decided to go with using pre-existing binary file to execute commands, what I mean by that is basically having a pre-installed tool (youtube-dl \u0026amp; pigz) on the system before using this application.\nIf the readme file of youtube-dl is checked, youtube-dl can be installed as command line tool into your environment. It means that we can call the tool whenever we need from our application. There are other ways to accomblish it as well, such as instead of using pre-existing binary file, we can implement the functions in our application. However, the main idea of this post is NOT about how to create or use the library, the main idea is to present how it easy to have automated way of retrieving Youtube playlist videos and saving to Github Releases. The other requirement regarding to compress can be used in similar way. (- using a pre-existin command from system -)\nTo make things simple and extendable (- which means in case of more integration of tools we should be able to accomblish it without changing, many lines of code -), I will generate a main Client struct which will have exec function and it will be overridden according to command we pass.\nThe main client struct :\ntype Client struct { //youtube-dl client YoutubeDL *YoutubeDL // Tar client Tar *Tar // Used to enable root command sudo bool // flags to service flags []string // enable debug or not debug bool // Implementation of ExecFunc. execFunc ExecFunc // Implementation of PipeFunc. pipeFunc PipeFunc } Client struct has some fields which enables us to override whenever we want, the struct contains exec(cmd string, args ...string) ([]byte, error), shellPipe(stdin io.Reader, cmd string, args ...string) ([]byte, error), and shellExec(cmd string, args ...string) ([]byte, error) functions. It can be extended according to our requirements in the future. The explanations of the functions are given on top of functions inside the source code.\nFor the youtube-dl client, I implemented only a function (-the client functionalities are really easy to extend-), which downloads all videos on given playlist by using pre-existing command line tool youtube-dl.\npackage client type YoutubeDL struct { c *Client } // exec executes an ExecFunc using \u0026#39;youtube-dl\u0026#39;. func (ytdl *YoutubeDL) exec(args ...string) ([]byte, error) { return ytdl.c.exec(\u0026#34;youtube-dl\u0026#34;, args...) } // DownloadWithOutputName generates Folder named with Playlist name // downloads videos under given playlist url to Folder func (ytdl *YoutubeDL) DownloadWithOutputName(folderName, url string) error { cmds := []string{\u0026#34;-o\u0026#34;, folderName + \u0026#34;/%(playlist_index)s - %(title)s.%(ext)s\u0026#34;, url} _, err := ytdl.exec(cmds...) return err } For any other additinal tool to use, it is extremely practical to add, for tar tool I have implemented following for specific purpose ( -which is compressing downloaded videos in paralel- ).\npackage client type Tar struct { c *Client } // exec executes an ExecFunc using \u0026#39;tar\u0026#39; command. func (tr *Tar) exec(args ...string) ([]byte, error) { return tr.c.exec(\u0026#34;tar\u0026#34;, args...) } // CompressWithPIGZ using tar with pigz compress program to compress given data func (tr *Tar) CompressWithPIGZ(fileName, folderToCompress string) error { cmds := []string{\u0026#34;--use-compress-program=pigz\u0026#34;, \u0026#34;-cf\u0026#34;, fileName, folderToCompress} _, err := tr.exec(cmds...) if err != nil { return err } return nil } Now, it is clear that for the both statements in the requirements section has been done. However, I wanted to keep track of what I have downloaded and release, for this reason, I have created two different csv files. They are called playlist-list.csv and old-playlist-list.csv under resources/ directory in the repository, playlist-list.csv will include all list of playlist URLs with preferred folder name to download. Futhermore, as you can guess, old-playlist-list.csv will include all the playlists which are downloaded and released. Once the playlist is downloaded and released with Github actions playlist-list.csv will be wiped and all content will be appended into old-playlist-list.csv file.\nIt will give easy way of checking what has been downloaded and released.\nThe code for reading and writing to csv files are pretty easy, and can be checked under main.go in the repository.\nWorkflow File The workflow file will include some steps, which are;\nInstall pigz : required to compress data in parallel. Install youtube-dl : required to download playlist from given URL. Build Binary : required to have combined binary which handles both download and compress using pre-existing tools on the system. Create Release : the step which initializes releases. Run Binary : executes the program Upload videos to Github releases : uploads downloaded content to releases. Remove playlist and append downloaded playlists to old list : updates the list inside the playlist file and commits on master branch. The steps given above are clickable to see inside the workflow on repository.\nThis small project is created for exclusive purpose, and it is very suitable to extend functionalities. However, there are many gaps regarding to the project such as;\nit does NOT check the given playlists whether they have been already released or not. it does NOT split created tar.gz files into 2 GB splits ( since it is required to have a file on Github releases under 2GB, but there is NO limitation for overall size of files on Github releases.) Does NOT have error handling mechanism and more. These are the points which appears when the project is checked at first glance, however there are more missing points which could be done. However, the main aim was to give idea how to accomblish automated way of downloading youtube videos and releasing with Github actions.\nI am personally using it for personal needs whenever I find useful playlist, I include it into playlist-list.csv file and pushing the changes by tagging the commit in semantic versioning format.\nThere are tons of other services which could be integrated such as Slack, Discord, Mail or any another notification systems and more, however, to keep the post short and do not bother you, it is enough for now as it is.\nThe rule for the workflow could be easily changed, like instead of running it in tagged commits, it can run in scheduled way by changing run condition only, as shown below.\nname: Download \u0026amp; Release Youtube Playlists on: schedule: - cron: \u0026#39;0 0 * * *\u0026#39; # it means every day at midnight the workflow will run If you require or would like to have more features, or fixes, suggestions and etc, you are more welcome to open issues.\nGithub Limitations Since we are using Github actions, we have some limitations regarding to usage of it.\nThe limitations regarding to file sizes in releases, according to Github Statement here:Distributing large binaries\nBasically, a file size which will be uploaded to releases should NOT exceeds 2 GB. However, keep in mind that it is per file, there is NO limitation for overall size of the release :). It means that repository will be updated to split files into chunks if size of the file exceeds 2 GB. So, in case of 15 GB of playlist, it should be uploaded in 2GB chunks to releases. (- a feature which is NOT exists on youtubeto yet -)\nThere are some more limitations:\nJob execution time - Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.\nWorkflow run time - Each workflow run is limited to 72 hours. If a workflow run reaches this limit, the workflow run is cancelled.\nMore details about limmitations on Github Actions: Usage Limits\nIt is good to keep in mind the given limitations above.\nJob execution time and Workflow run time can be easily fixed if you have your own server.\nIf you would like to run Github Actions in your server, there is no limitation regarding to Job execution time and Workflow run time.\nCheck out how to setup Github Actions for your server from here:\nSetup self hosted runners\nRepository youtubeto: Automated Youtube PlayList Releaser\nDemo ","permalink":"https://mrturkmen.com/posts/download-release-youtube-playlists/","summary":"\u003cp\u003eIn some moments, Youtube algorithm is working perfect, but sometimes it shows a video from ten years ago from nowhere. For the moments where it shows and suggests videos/playlists to us, we might want to save the list of playlist and watch in some other time. It could be on a plane, train, bus, whenever you are planning to spent some time. However, taking the URL of a playlist and saving it to your cute note program might not be sufficient enough.\nThere is a high chance that it will be forgotten or missed, therefore, I thought that it would be nice to have an automated way of saving playlists on somewhere and download them when I need. (- in particular, when there is no or limited internet connection -).\u003c/p\u003e","title":"youtubeto: download and save playlists to releases on Github"},{"content":" In this post, I will be describing to setup a workflow to build and release your Latex files through Github actions. First of all, keep in mind that this post is not about what is Latex and how to use it.\nIt is extremely nice to integrate daily development tools such as CI/CD to your preparation of paper, without any hassle. Why is that because it is cool to track of what has been changed on a paper over time. In fact, having a couple of people who are responsible in different parts of paper, sometimes blocks others. Therefore, having such a workflow will increase productivity for everyone in a group. Whenever pull request created to main branch, it will be easy to check typos, logic errors and missing points by others.\nLatex preparation Setup Github Actions Proof of Concept Latex preparation I am assuming that you have agreed to work on Latex template to complete a paper. In this case, there is only small step left to do, create a Github repository (-it should be on Github, Github Actions will be used-) and push all files of your Latex template. (-in general, in following structure-)\n|-sections | introduction.tex | related_works.tex | problem.tex | solution.tex | conclusion.tex |- main.tex |- references.bib The given example structure can be changed according to your wishes, however important and logical part is that having main.tex on root directory of repository.\nOnce it is set, there is only one step to complete which is setting up Github Action workflows.\nSetup Github Actions There are a few different Github Actions to use for compiling Latex document to PDF on marketplace. Most preferred one is https://github.com/xu-cheng/latex-action and it is quite easy to integrate and use.\nIt basically creates generated PDF file from provided Latex file, it can be set in workflow file as given below: (- Note that this workflow runs on tagged commits which has a tag with *.*.* pattern -)\nname: Build LaTeX document on: tags: - \u0026#39;*.*.*\u0026#39; # semantic versioning jobs: build_latex: runs-on: ubuntu-latest steps: - name: Set up Git repository uses: actions/checkout@v2 - name: Compile LaTeX document uses: xu-cheng/latex-action@v2 with: root_file: main.tex However, setting up only this job is not sufficient enough to have completed workflow, we require to more jobs which are Create Release and Upload Release. As you may guess from their name, first one will create the release and second one will upload provided file to releases page. It can be setup as following\nname: Release Compiled PDF on: push: tags: - \u0026#39;*.*.*\u0026#39; jobs: build_latex: runs-on: ubuntu-latest steps: - name: Set up Git repository uses: actions/checkout@v2 - name: Compile LaTeX document uses: xu-cheng/latex-action@v2 with: root_file: main.tex - name: Create Release id: create_release uses: actions/create-release@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: tag_name: ${{ github.ref }} release_name: Release ${{ github.ref }} draft: false prerelease: false - name: Upload Release Asset id: upload-release-asset uses: actions/upload-release-asset@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: upload_url: ${{ steps.create_release.outputs.upload_url }} asset_path: ./main.pdf asset_name: main.pdf asset_content_type: pdf The given workflow is completed version of what you might have at the end. In summary, it builds PDF from provided Latex file, creates release and upload file to release. For more details, you can check information on each action page.\nProof of Concept Here is example repository to check completed version.\nhttps://github.com/merkez/latex-on-ci-cd\n","permalink":"https://mrturkmen.com/posts/build-release-latex/","summary":"\u003chr\u003e\n\u003cp\u003eIn this post, I will be describing to setup a workflow to build and release your Latex files through Github actions. First of all, keep in mind that this post is not about what is Latex and how to use it.\u003c/p\u003e\n\u003cp\u003eIt is extremely nice to integrate daily development tools such as CI/CD to your preparation of paper, without any hassle. Why is that because it is cool to track of what has been changed on a paper over time. In fact, having a couple of people who are responsible in different parts of paper, sometimes blocks others. Therefore, having such a workflow will increase productivity for everyone in a group.  Whenever pull request created to main branch, it will be easy to check typos, logic errors and missing points by others.\u003c/p\u003e","title":"auto-latex: generate and handle latex through github actions"},{"content":"In this post, I am going to write demo for a tool which I have just met, it is called Evans. It is basically universal gRPC client. What it means ? Basically when you have gRPC server and would like to test gRPC calls without creating client, you can test server side calls with Evans. It is known that gRPC is very common communication method between microservices, it can be used for internal and external communication. I do not have intention to explain what gRPC is in this post since it is not the purpose. If required documentation of gRPC can be investigated.\nCreate a simple gRPC server To demonstrate and see how evans works, a running gRPC should be exists, for this reason, I am going to provide a sample gRPC server. For this purpose, I will use Go programming language, however gRPC is supporting more programming languages which you may more familiar than Go.\ngRPC server is basically an API endpoint where clients can make requests, since it is an API, first thing could be to define which methods will be used for this service.\nFor simplicity and purpose of this post, I have created a basic microservice which will has four calls namely, Add,Delete,List and Find. Since the purpose is to understand how evans works, the gRPC server does not need to be complex or includes lots of calls.\nSample repository A sample microservice is created for demonstration purposes and all codes are available here : https://github.com/merkez/BookShelf\nIf you have already a running gRPC server, you can direcly pass to demonstration of evans, if not, you can clone https://github.com/merkez/BookShelf repository and test evans out.\nDefining calls In my opinion, it is always nice to prepare proto file before hand, because it is like a contract which creates your main service. Let\u0026rsquo;s imagine you would like to add, delete, list and find the books that you have read or wish to read. For this purpose, microservice should have at least four different calls which are Add, Delete, List and Find. There is no limit to have more calls however in order to do not get out of topic, I am keeping it small.\nFollowing proto file would be enough for BookShelf service.\n// it is important to declare syntax version syntax = \u0026#34;proto3\u0026#34;; service BookShelf { rpc AddBook(AddBookRequest) returns (AddBookResponse) {} rpc ListBook (ListBooksRequest) returns (ListBooksResponse) {} rpc DelBook (DelBookRequest) returns (DelBookResponse){} rpc FindBook (FindBookRequest) returns (FindBookResponse){} } message AddBookRequest { BookInfo book = 1; message BookInfo { string isbn =1; string name =2; string author=3; string addedBy=4; } } message AddBookResponse { string message = 1; } message ListBooksRequest { // no need to have anything // could be extended to list books based on category ... } message ListBooksResponse { repeated BookInfo books =1; message BookInfo { string isbn =1; string name =2; string author=3; string addedBy=4; } } message DelBookRequest { string isbn =1; } message DelBookResponse { string message =1; } message FindBookRequest { string isbn =1; } message FindBookResponse { Book book = 1; message Book { string isbn =1; string name =2; string author=3; string addedBy=4; } } Once proto file is declared, it becomes more easy to continue. For the demostration purposes, I will store information of books in memory. However, as you know, it is NOT acceptable for any production level application.\nCompile Proto file proto files are great since once you have defined what you need, you can directly generate codes in available languages which are represented in gRPC supported languages. The generation of codes in your desired language is pretty straitforward, I am going to generate the code for Go programming language.\n$ protoc -I proto/ proto/bs.proto --go_out=plugins=grpc:proto It will generate ready to use Go source code for your rpc calls which are defined in proto file.\nAfterwards, necessary piece of codes should be implemented, which are in memory store and book struct. For the aim of this post, I assumed that you have created all rest of the code as given in example gRPC server (-BookShelf-).\nNOTE: Generating source code through protoc requires to have protoc tool to be installed before hand. Installation of protoc is over here\nRun gRPC server The post is not covering all aspects of gRPC, proto buffers, Go language and those are not the intention of this post. Therefore, I am assuming that you had gRPC server and would like to test out and see whether your proto contract is running correctly without creating client side codes. Once it is confirmed that your gRPC calls are running without encouraging any unseen problems, then creating client side code will be much easy without any problem.\nYou can start gRPC server with:\n$ go run server/main.go BookShelf gRPC server is running .... Once gRPC server is up and running, you can use evans tool for inspecting gRPC server for available inquires.\nDemonstration of EVANS Evans is an open source project which is available at Github and I found it pretty useful, in particular, for people who have no idea what kind of calls are available in proto file, as it states its explanation, it is universal gRPC client. Installation of evans and more information is given its readme file.\nIt has plenty of features which are very handy to use for automating and testing some stuff on top of existing gRPC server or new one.\nLet\u0026rsquo;s make a demo, when you are using evans your gRPC server should be up and running, in order to make communication with universal gRPC client - evans. I assumed that you have followed readme file of evans and installed it correctly.\nNote that port 9000 is given because it is port of gRPC server.\n❯ evans -r -p 9000 ______ | ____| | |__ __ __ __ _ _ __ ___ | __| \\ \\ / / / _. | | \u0026#39;_ \\ / __| | |____ \\ V / | (_| | | | | | \\__ \\ |______| \\_/ \\__,_| |_| |_| |___/ more expressive universal gRPC client BookShelf@127.0.0.1:9000\u0026gt; show services +-----------+----------+------------------+-------------------+ | SERVICE | RPC | REQUEST TYPE | RESPONSE TYPE | +-----------+----------+------------------+-------------------+ | BookShelf | AddBook | AddBookRequest | AddBookResponse | | BookShelf | ListBook | ListBooksRequest | ListBooksResponse | | BookShelf | DelBook | DelBookRequest | DelBookResponse | | BookShelf | FindBook | FindBookRequest | FindBookResponse | +-----------+----------+------------------+-------------------+ BookShelf@127.0.0.1:9000\u0026gt; As you can observe all calls which are used in proto file can be used through evans, moreover its usage is pretty straitforward.\nYou can check demonstration video below.\n⚠️ You may wish to change the video quality to 1080p60\nIf you have any questions, fix, or something else, do not hesitate to contact with me.\n","permalink":"https://mrturkmen.com/posts/grpc-calls-with-evans/","summary":"\u003cp\u003eIn this post, I am going to write demo for a tool which I have just met, it is called \u003cstrong\u003eEvans\u003c/strong\u003e. It is basically universal gRPC client. What it means ? Basically when you have gRPC server and would like to test gRPC calls without creating client, you can test server side calls with \u003cstrong\u003eEvans\u003c/strong\u003e. It is known that gRPC is very common communication method between microservices, it can be used for internal and external communication. I do not have intention to explain what gRPC is in this post since it is not the purpose.\nIf required \u003ca href=\"https://grpc.io/docs/what-is-grpc/introduction/\"\u003edocumentation of gRPC\u003c/a\u003e can be investigated.\u003c/p\u003e","title":"evans: universal gRPC client demonstration"},{"content":"In recent post, which is Setup Highly Available Kubernetes Cluster with HAProxy , a highly available Kubernetes cluster is created. However, once I started to dig in and deploy some stuff to cluster, I realized that I am not able to connect any deployed application or services. For instance, when an web application is deployed using HAProxy load balancer (endpoint), and check from kubectl (on client side), its status is running. However, that application could not be reached from outside world although I re-patch an external IP address by following command\n$ kubectl patch svc \u0026lt;application-name\u0026gt; -n \u0026lt;name-of-namespace\u0026gt; -p \u0026#39;{\u0026#34;spec\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;LoadBalancer\u0026#34;, \u0026#34;externalIPs\u0026#34;:[\u0026#34;\u0026lt;haproxy-ip-address\u0026gt;\u0026#34;]}}\u0026#39; After some searching and reading, I realized that worker nodes require their own ingress controllers in order to forward traffic between them in case of load. I will be giving more information of how I fix the issue, however let\u0026rsquo;s learn some basic terms and general information about ingress controller.\nWhat is ingress controller ? The best and simple explanation to this question is coming from Kubernetes official documentation over here, as they are expressing that ;\nIngress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.\nAn Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontend to help handle the traffic.\nWhenever you have services which are running inside a cluster and would like to access them, you need to setup ingress controller for that cluster. The missing part was having no ingress controller on worker nodes in my k8s cluster. Everything was working however there was no access to them from outside world, that\u0026rsquo;s why ingress controller should take place in cluster architecture.\nIn this post, I will go for NGINX ingress controller with its default setup, however there are plenty of different ingress controllers which you may go for. I might change NGINX to Traefik in future but it depends on requirements yet for now, I will go with nginx ingress controller. The reason is that, it is super easy to setup, super rich with different features, included Kubernetes official documentation and fulfill what I am expecting for now.\nUpdates to cluster Let\u0026rsquo;s briefly what I have explained in previous post;\nCreate VMs Setup SSH connection Use KubeSpray to deploy cluster Create HAProxy and establish SSH connection with all nodes. I have noticed that when deploying cluster, some add-ons should be enabled in order to use ingress controller from cluster with external HAProxy load balancer. Now, since cluster deployment was established with Ansible playbooks, it is not needed to setup everything from scratch. All modified configuration can be re-deployed without effecting any resource which is exists on cluster setup. It means that, I can enable required parts in configuration file and re-deploy cluster as I did on previous post.\nEnable ingress controller from inventory file inside KubeSpray\n$ vim inventory/mycluster/group_vars/k8s-cluster/addons.yml # Nginx ingress controller deployment ingress_nginx_enabled: false -\u0026gt; true Once this configuration part is updated from existing KubeSpray configuration files, k8s cluster should be redeployed with same command in previous post\nAssumption : previous configured KubeSpray settings are used.\n$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml It will take a while and update all necessary parts which are required.\nInclude Ingress API object to route traffic from external HAProxy server to internal services\nTo include Ingress API object, HAProxy configuration file should be modified, following lines should be added to /etc/haproxy/haproxy.cfg file.\n$ vim /etc/haproxy/haproxy.cfg frontend kubernetes-ingress-http bind *:80 default_backend kubernetes-worker-nodes-http backend kubernetes-worker-nodes-http balance leastconn option tcp-check server worker1 10.0.128.81:80 check fall 3 rise 2 server worker2 10.0.128.137:80 check fall 3 rise 2 server worker3 10.0.128.156:80 check fall 3 rise 2 In given configuration balancing algorithm is leastconn which can be changed into any load balancer algorithm which is supported by HAProxy, however leastconn algorithm is fitting more to what I would like to achieve that\u0026rsquo;s why it is declared as leastconn. Note that this configuration addition is on top of added part on previous post.\nOnce HAProxy configuration is updated, HAProxy should be restarted systemctl restart haproxy. It is all for HAProxy configuration, now let\u0026rsquo;s dive into setting up NGINX Ingress Controller.\nSetup NGINX Ingress Controller It is super simple to deploy and setting up NGINX ingress controller since it is well documented and explains required parts in detail. To setup NGINX Ingress Controller, I will follow official guideline which is exists on NGINX Ingress Controller Installation.\nImage is taken from (https://www.nginx.com/products/nginx/kubernetes-ingress-controller/#resources)\nIn normal cases, the situation is as given figure above, however, since in existing k8s cluster, I am using HAProxy for communicating with clients, I need NGINX ingress controller inside worker nodes which will manage running applications/services by communicating with HAProxy and eventually, the services will be accessible from outside world.\nIf I summarize how overview diagram will look like in my case is like in given figure below.\nIt can be observed that, in given k8s cluster overview, HAProxy is in front, it communicates with clients, afterwards transmitting request based on defined rule on HAProxy configuration. Each worker node has NGINX ingress controller, what exactly it means, whenever a request appear to cluster, worker nodes will agree between each other and response back to user without having any problem. Since NGINX ingress controller is capable of load balancing inside worker nodes as well.\nThere is also Ingress Resource Rules part inside cluster, what it does, is that all routing rules based on path forwarded given service, an example on this is given below.\nSteps to create NGINX Ingress controller All steps shown below for installation of NGINX Ingress Controller taken from https://docs.nginx.com/nginx-ingress-controller/installation/\nMake sure that you are a client with administrator privilege, all steps related to NGINX ingress controller should be done through kubectl (on client computer/server)\nClone Ingress Controller Repo $ git clone https://github.com/nginxinc/kubernetes-ingress/ $ cd kubernetes-ingress Create a namespace and a service account for the Ingress controller $ kubectl apply -f common/ns-and-sa.yaml Create a cluster role and cluster role binding for the service account $ kubectl apply -f rbac/rbac.yaml Create a secret with a TLS certificate and a key for the default server in NGINX $ kubectl apply -f common/default-server-secret.yaml Create a config map for customizing NGINX configuration: $ kubectl apply -f common/nginx-config.yaml Afterwards, there are two different ways to run NGINX ingress controller deployment, which are as daemonset or as deployment. Main difference between those are summarized on official installation page as;\nUse a Deployment. When you run the Ingress Controller by using a Deployment, by default, Kubernetes will create one Ingress controller pod.\nUse a DaemonSet: When you run the Ingress Controller by using a DaemonSet, Kubernetes will create an Ingress controller pod on every node of the cluster.\nI will go with DaemonSet approach, the reason is that generally when you have background-ish tasks which will run non-stateless then DaemonSet is more preferred way of running it.\n$ kubectl apply -f daemon-set/nginx-ingress.yaml Once it is applied as daemon set, the result could be checked with following command and result will be similar to given result below.\n$ kubectl get all NAME READY STATUS RESTARTS AGE pod/nginx-ingress-47z8r 1/1 Running 0 24h pod/nginx-ingress-cmkfq 1/1 Running 0 24h pod/nginx-ingress-ft5pv 1/1 Running 0 24h pod/nginx-ingress-q554l 1/1 Running 0 24h pod/nginx-ingress-ssdrj 1/1 Running 0 24h pod/nginx-ingress-t9jml 1/1 Running 0 24h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/nginx-ingress 6 6 6 6 6 \u0026lt;none\u0026gt; 24h Deploy Example Application To test how an application will be exposed to externally from k8s cluster, an example applicaton could be deployed as given below. Note that the following example is simplest example for this context, hence, keep in mind that it might require more configuration and detailed approach then described here when you would like to deploy more complex applications.\nCreate a sample NGINX Web Server (Using provided example) nginx-deploy-main.yml\napiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 2 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 Taken from https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/\nIn given yaml deployment file above, two replicas of NGINX:1.14.2 will be deployed to cluster and it has name of nginx-deployment. The yaml explains itself very well.\nIt can be deployed either through directly from official link or from your local depends on your preferences.\n$ kubectl apply -f https://k8s.io/examples/application/deployment.yaml ## or you can do same thing with local file as given below $ kubectl apply -f nginx-deploy-main.yml Expose deployment:\n$ kubectl expose deploy nginx-deployment --port 80 Once it is deployed to cluster and exposed, there is one step left for this simple counter example is that, exposing the service and creating ingress rule (resource) in yaml file, by specifiying kind as Ingress.\napiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-ingress spec: rules: - host: \u0026lt;dns-record\u0026gt; (a domain like test.mydomain.com) http: paths: - path: / backend: serviceName: nginx-deployment servicePort: 80 The crucial part is serviceName and servicePort which are defining specifications of the services within cluster. The yaml specifications can be expanded as shown below, assume that you have wildcard record in your domain name server and have multiple services which are running in same port in a cluster, yaml file can be re-defined as given below.\nnginx-ingress-resource.yml\napiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-controller spec: rules: - host: \u0026lt;dns-record\u0026gt; (a domain like test.mydomain.com) http: paths: - path: / backend: serviceName: nginx-deployment servicePort: 80 - path: /apache backend: serviceName: apache-deployment servicePort: 80 - path: /native-web-server backend: serviceName: native-web-server-deployment servicePort: 80 Keep in mind that all given services should be deployed before hand otherwise when a request made to any path which is not deployed, it may return either 404 or 500. There are plenty of different options to define and update the components in a k8s cluster. Therefore, all yaml files should be changed according to requirements.\nCreate ingress controller rules from provided yaml file\n$ kubectl create -f nginx-ingress-resource.yml Now, the NGINX web server deployment is ready on given DNS record in yaml file and according to request paths different services can be called which are also running inside kubernetes cluster.\nNote that, provided yaml files are just simple example of deploying NGINX web server without any certification, when certificates (HTTPS) enabled or any other type of deployment happened different configurations should be applied.\nWhen everything goes without any problem, you will have a cluster which uses NGINX Ingress controller for internal cluster routing and HAProxy as communication endpoint for clients. Keep in mind that whenever a new service or deployment take place, required configuration should be enabled in HAProxy configuration as it is enabled for port 80 applications above. Different services will have different requirements therefore it is important to catch main logic in a setup. It is all done for this post.\nCheers !\n","permalink":"https://mrturkmen.com/posts/setup-ingress-controller/","summary":"\u003cp\u003eIn recent post, which is \u003ca href=\"https://mrturkmen.com/install-ha-kubernetes-cluster/\"\u003eSetup Highly Available Kubernetes Cluster with HAProxy \u003c/a\u003e, a highly available Kubernetes cluster is created. However, once I started to dig in and deploy some stuff to cluster, I realized that I am not able to connect any deployed application or services. For instance, when an web application is deployed using HAProxy load balancer (endpoint), and check from \u003ccode\u003ekubectl\u003c/code\u003e (on client side), its status is running. However, that application could not be reached from outside world although I re-patch an external IP address by following command\u003c/p\u003e","title":"haproxy-with-nginx: setting them up for k8s cluster"},{"content":"The main purpose of this blog post a simple walkthrough of setting up Kubernetes cluster with external HAProxy which will be the endpoint where our kubectl client communicates over. Node specifications for this setup is given as shown in the table below. Keep in mind that all of them has access to each other with password and without password. The environment which Kubernetes cluster will stay is running on OpenStack. It means that once a configuration (ssh keys, hosts, and etc) is done for example master 1 then all other nodes could be initialized through snapshot of master 1. To be able to setup such a Kubernetes cluster easily, I will be using KubeSpray which is a repository where it has all required configuration and playbooks for setting up necessary cluster.\nNode Specification Prerequisites General Overview KubeSpray Configuration External Load Balancer Setup (HAProxy) Setup KubeSpray Configuration The intention of this walkthrough is that setting up your own Kubernetes cluster in your own servers, this post is not very useful for people who are already using cloud provider solutions.(Kubernetes cluster as a service). You can checkout following resources listed below : (few of them :) )\nCloud Providers Solutions:\nAzure Kubernetes Service - AKS Google Kubernetes Engine - GKE Managed Kubernetes on DigitalOcean Kubernetes on AWS Node Specification Kubernetes cluster will be setup on following nodes in the table below, note that HAProxy will run on another node and all ansible playbooks and setting up Kubernetes cluster will be managed through HAProxy. Keep in mind that all nodes + HAProxy is under same subnet internally which means that we will only one external IP address where HAProxy use and kubectl clients communicate. All instances are running on ubuntu_18.04, it means that the instructions and steps may not work with another system.\nPrerequisites Nodes Requirements of KubeSpray Setting up SSH Key Across Nodes Getting snapshot ( -it is optional -) Setting up login with password General Overview The following sketch is general overview of how Kubernetes cluster will look like at the end of this walkthrough, the figure is super overviewed version of cluster.\nIn given figure above, nodes do not have any external IP adress however, including HAProxy, all of them in same subnet, only HAProxy has external IP address which will be reachable by kubectl clients.\nBefore moving installation step of Kubernetes cluster, we need to setup a sample master node (instance) with predefined configuration. Since we will have only one server which is open to outside world, we need to make sure that there is a connection between HAProxy and sample master node. I am currently calling it sample master node, it is because, preliminary configurations such as authentication with password, disabled swap area and ssh keys will be all configured. This sample master node should be started and accesible over HAProxy, which means that in order to access to sample master node, I should do following;\nSSH to HAProxy using SSH key (Password Login disabled) like ssh -i ~/.ssh/id_rsa \u0026lt;username\u0026gt;@\u0026lt;ha-proxy-external-ip\u0026gt; Copy SSH Key to HAProxy, which let you in to sample master node Then SSH to sample master node with same approach. (ssh ~/.ssh/masternode.pem \u0026lt;username\u0026gt;@\u0026lt;master-node-ip\u0026gt; After you are inside sample master node, now, some configurations and setting should be done. Afterwards, we can initialize other five nodes from snapshot of configured sample master node.\nSteps:\nEnable Password Login if not enabled already. $ echo \u0026#34;PermitRootLogin yes\u0026#34; \u0026gt;\u0026gt; /etc/ssh/sshd_config $ sed -i -E \u0026#39;s/PasswordAuthentication no/PasswordAuthentication yes/g\u0026#39; /etc/ssh/sshd_config Specify Password for ROOT $ sudo su $ passwd Given commands will ask new unix password for root user. Define the password and do not forget or lose it. Since we will gonna use snapshot of this configured machine, all settings will be same, I did like that to shortcut the process.\nDisable swap area (RUN ALL COMMANDS AS ROOT) $ swapoff -a Afterwards, exit from sample master node, create snapshot of that node (it is called volume snapshot in OpenStack), once you have successfully created snaphot, all five other nodes should be initialized from snapshot of this sample master node. This way, there is no need to repeat same steps described above.\nIn case of not having possibility to create snapshot follow given steps (if and if only, you could NOT create snapshot and initialize other five nodes from the snapshot)\nCreate all nodes (workers and masters)\nEnable SSH connection to all nodes from HAPRoxy server.\nFrom HAProxy server, execute following steps. (-Make sure that you have configured SSH connection with ROOT priviledges and have access to all nodes from HAProxy node -)\nOnce you are sure that you have SSH access to all nodes from HAProxy through SSH, implement following steps.\nInstall parallel-ssh (-to run a command in parallel on nodes-) (run with ROOT priviledges) $ apt-get update \u0026amp;\u0026amp; apt-get install -y pssh Install HAProxy (as ROOT priviledges) $ apt-get install -y haproxy Modify /etc/hosts (-For easy communication through nodes-) Append worker and master node IPs to /etc/hosts file\n$ vim /etc/hosts 10.0.128.156 worker3 10.0.128.137 worker2 10.0.128.81 worker1 10.0.128.184 master3 10.0.128.171 master2 10.0.128.149 master1 Create nodes text file on home directory $ cat nodes worker3 worker2 worker1 master3 master2 master1 Since IP addresses of them defined in /etc/hosts file, system can now recognize and connect IPs of them through just by name\nGenerate and Copy SSH Key to all nodes (Required for easy communication) If there is already a SSH key (like in ~/.ssh/id_rsa), you can use it as well.If not, you can do following step\n$ ssh-keygen # will prompt passphrase, you can leave empty , NOTE THAT IF YOU DO NOT HAVE SSH KEY, GENERATE IT. $ for i in $(cat nodes); ssh-copy-id $i; done The for loop given as second command will copy ssh key to all nodes, then accesing any node without password will be flawless.Like given command below;\n$ ssh master1 # in defualt uses same username with terminal session Disable swap area on all nodes (Note that if you are using snapshot method, no need to do this step) $ parallel-ssh -h nodes -i \u0026#34;swapoff -a\u0026#34; Parallel SSH tool is handy to complete tasks in parallel for multiple hosts.\nKubeSpray Configuration KubeSpray is a repository to setup Kubernetes clusters with predefined configuration settings using Ansible playbooks. The usage of KubeSpray is pretty straightforward, as default settings, KubeSpray is using internal load balancers in each worker node, which means that when you setup a Kubernetes cluster using default values of KubeSpray, you will have following arch overview.\nHowever, in this guide, external load balancer approach will be used to setup cluster, if you wish to leave everything as default with KubeSpray, you can skip this External Load Balancer Setup part.\nExternal Load Balancer Setup (HAProxy) Modify configuration file of HAProxy to enable external LoadBalancer, copy this following configuration and append to /etc/haproxy/haproxy.cfg. (end of file)\nlisten kubernetes-apiserver-https bind \u0026lt;your-haproxy-internal-ip\u0026gt;:8383 mode tcp option log-health-checks timeout client 3h timeout server 3h server master1 \u0026lt;your-master1-ip\u0026gt;:6443 check check-ssl verify none inter 10000 server master2 \u0026lt;your-master2-ip\u0026gt;:6443 check check-ssl verify none inter 10000 server master3 \u0026lt;your-master3-ip\u0026gt;:6443 check check-ssl verify none inter 10000 balance roundrobin Balance algorithm is roundrobin however you can change it from list of available balance algorithms provided by HAProxy.\nOnce it is done, save and restart HAProxy service.\n$ systemctl restart haproxy Setup KubeSpray Configuration Since external load balancer will be used, there is few things to be done to change default values in KubeSpray. Following steps will be done on HAProxy node.\nClone the project and prepare environment $ git clone https://github.com/kubernetes-sigs/kubespray $ apt-get install -y python3-pip # install pip3 if not installed $ cd kubespray Follow the guide on KubeSpray README.md file Following instructions taken from KubeSpray README.md\n# Install dependencies from ``requirements.txt`` sudo pip3 install -r requirements.txt # Copy ``inventory/sample`` as ``inventory/mycluster`` cp -rfp inventory/sample inventory/mycluster # Update Ansible inventory file with inventory builder declare -a IPS=(10.0.128.149 10.0.128.171 10.0.128.184 10.0.128.81 10.0.128.137 10.0.128.156) CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]} Modify generate hosts YAML file When you check inventory/mycluster/hosts.yaml file, you will notice that it created two master nodes, which we require three, add missing one properly to that list as shown below.\nall: hosts: master1: ansible_host: 10.0.128.149 ip: 10.0.128.149 access_ip: 10.0.128.149 master2: ansible_host: 10.0.128.171 ip: 10.0.128.171 access_ip: 10.0.128.171 master3: ansible_host: 10.0.128.184 ip: 10.0.128.184 access_ip: 10.0.128.184 worker1: ansible_host: 10.0.128.81 ip: 10.0.128.81 access_ip: 10.0.128.81 worker2: ansible_host: 10.0.128.137 ip: 10.0.128.137 access_ip: 10.0.128.137 worker3: ansible_host: 10.0.128.156 ip: 10.0.128.156 access_ip: 10.0.128.156 children: kube-master: hosts: master1: master2: master3: kube-node: hosts: master1: master2: master3: worker1: worker2: worker3: etcd: hosts: master1: master2: master3: k8s-cluster: children: kube-master: kube-node: calico-rr: hosts: {} Once it is done, the other thing which should be modified to use external load balancer HAProxy, is all.yaml file located under inventory/mycluster/group_vars/all/.\nall.yml is general configuration file which specifies main configurations of your cluster, it uses Nginx load balancer by default which means that each worker node has its own local nginx load balancer as given second figure above. If not specified anything else.\nDisable default load balancer $ vim inventory/mycluster/group_vars/all/all.yml loadbalancer_apiserver_localhost: false Add external load balancer HAProxy. $ vim inventory/mycluster/group_vars/all/all.yml ## External LB example config apiserver_loadbalancer_domain_name: \u0026#34;\u0026lt;domain-name-of-lb\u0026gt;\u0026#34; loadbalancer_apiserver: address: 10.0.128.193 port: 8383 Initialize cluster deployment # under kubespray/ directoy $ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml It will take around 10-15 minutes which depens on your cluster and if everything goes well, at the end of deployment through Ansible you will not face with any problem. If so, you can test it by SSH to master node and try kubectl cluster-info.\n$ kubectl cluster-info Kubernetes master is running at ..... To further debug and diagnose cluster problems, use \u0026#39;kubectl cluster-info dump\u0026#39;. It means that Kubernetes cluster with three master and three worker nodes available to use.\nNote that the default configuration of cluster could be changed more however before attempting to change default configuration, make sure that you did correct research on what to change on KubeSpray default settings. Otherwise, there might be problems regarding to customized configuration settings.\nFor more information stay updated and watch KubeSpray regarding to issues, pitfalls and more.\nLast step for this post is creating kubectl configuration for your personal/work computer to access the cluster. Install kubectl on your environment. Afterwards copy configuration from master node to your ~/.kube/ as config.\nSince we have only one endpoint, configuration file should be copied to HAProxy Server then your computer, through rsync or scp\nOn HAProxy Server $ scp root@master1:/etc/kubernetes/admin.conf config # will copy admin.conf as config $ cp config /home/ubuntu/ # copy to a user home dir $ chown ubuntu:ubuntu /home/ubuntu/config # change owner of the file On your personal/work computer $ scp -i ~/.ssh/haproxy.pem ubuntu@\u0026lt;ha-proxy-ip\u0026gt;:/home/ubuntu/config ~/.kube/ Now, you should be able to get and dump your cluster information as in master nodes.\n$ kubectl cluster-info Kubernetes master is running at ..... To further debug and diagnose cluster problems, use \u0026#39;kubectl cluster-info dump\u0026#39;. There are lots of configurations and different settings regarding to Kubernetes cluster environment and generally using Cloud Provider solutions are less painful or painless. However, sometimes it is less costly to setup your own environment and having full access to anything could be better for learning under the hood things or creating highly customized environments. It really depends on your situation therefore it is up to you to go and setup your own Kubernetes cluster or use it as service from cloud providers.\nBy the way, thanks for giving time to checkout the post 😉\n","permalink":"https://mrturkmen.com/posts/install-ha-kubernetes-cluster/","summary":"\u003cp\u003eThe main purpose of this blog post a simple walkthrough of setting up Kubernetes cluster with external \u003ca href=\"http://www.haproxy.org/\"\u003eHAProxy\u003c/a\u003e which will be the endpoint where our \u003ccode\u003ekubectl\u003c/code\u003e client communicates over.\nNode specifications for this setup is given as shown in the table below. Keep in mind that all of them has access to each other with password and without password. The environment which Kubernetes cluster will stay is running on OpenStack. It means that once a configuration (ssh keys, hosts, and etc) is done for example master 1 then all other nodes could be initialized through snapshot of master 1. To be able to setup such a Kubernetes cluster easily, I will be using \u003ca href=\"https://github.com/kubernetes-sigs/kubespray\"\u003eKubeSpray\u003c/a\u003e  which is a repository where it has all required configuration and playbooks for setting up necessary cluster.\u003c/p\u003e","title":"haproxy: setting it up for highly available k8s cluster"},{"content":"VPN Kuralım Bugün sizlere kendinize ait VPN sistemi nasıl kurulur, onu anlatmak istiyorum, daha önce İngilizce olarak, yayınladım fakat Türkçe bir kaynağın da faydalı olabileceğini düşündüm. Burada anlatılanlar, ubuntu ailesine (16.04,18.04) ait sunucular üzerinde test edilmiştir.\nİlk olarak bulut hizmeti sağlayan bir şirketten bu DigitalOcean, Google Cloud, Microsoft Azure veya Amazon olabilir, sunucu kiralıyorsunuz, en ucuzu ve makul olanı DigitalOcean tarafından sunulan aylık 5 dolar olan sunucu diyebilirim. Sunucuyu kiraladıktan ve ssh bağlantısını sağladıktan sonra VPN kurulumuna geçebiliriz.\nVPN hakkında tam bilgisi olmayan arkadaşlar için şu şekilde özetlenebilir, sizin için oluşturulmuş sanal bir bağlantı noktası gibi düşünebilirsiniz. Yani VPN\u0026rsquo;e bağlandıktan sonra bilgisayarınızdan çıkan ve bilgisayarınıza gelen ağ trafik şifrelenmiş olarak işlenir. Üçüncü parti yazılımların veya MITM gibi saldırıların önüne geçmiş olursunuz.\nNeden kendi VPN sistemi kurmalıyız ? Çünkü şu anda var olan bütün VPN sistemleri, ücretsiz olarak hizmet sağlasa dahi, sizin bilgilerinizin satılması, arşivlenmesi ve gerektiğinde ilgili birimlere aktarılması amacıyla kaydedilmektedir. Bunun ne gibi zararları olabilir gelin birlikte şöyle bir sıralayalım:\nOltalama saldırılarına sadece sizin bilebileceğiniz bilgiler ile maruz kalma. Ziyaret ettiğiniz siteler tarafından reklam bombardımanına maruz kalma. Kişisel bilgilerinizin reklam veren ajanslara satılması, bu durum birçok kişi tarafından tam olarak anlaşılamıyor, yani şu şekilde anlaşılamıyor, internet üzerinden alışveriş yapan A kişisi, kendine ait bilgilerin, onun bilgilerini satacak kişiler tarafından değersiz olduğuna inanıyor ve hiçbir gizlilik sağlamadan internet kullanımına devam ediyor. Bu sonunda o kişiye zarar vermese bile o kişinin konuştuğu, görüştüğü veya birlikte çalıştığı arkadaşlara zarar verebiliyor. Burada sıralananlar sadece buzdağının görünen ucu bile diyemeyiz, günümüzde veri işleme teknikleri ve yaklaşımları öyle gelişmiştir ki siz bile kendinize ait olan bir şeyin varlığına farkında olmadan onlar işlemleri tamamlamış oluyor :).\nBu ve bunlardan çok daha fazla nedenden dolayı VPN kullanımı şart diyebilirim. Peki bunu nasıl yapacağız, bu kısımdan sonra sizin bir bulut sağlayıcısı tarafından sunucunu kiraladığınızı ve ssh bağlantısını sağladığınızı varsayıyorum.\nBu gönderide WireGuard VPN uygulaması kullanılacaktır. WireGuard VPN uygulaması açık kaynaklı bir uygulama olup, sağladığı imkanlar sayesinde diğer VPN uygulamalarına (OpenVPN ve diğerleri) kıyasla çok daha hızlı ve güvenilirdir.\nSunucu Ayarları VPN uygulamasını kiraladığımız sunucu üzerine kuralım.\n$ sudo apt-get update \u0026amp;\u0026amp; sudo apt-get upgrade -y $ sudo add-apt-repository ppa:wireguard/wireguard $ sudo apt-get update $ sudo apt-get install wireguard Uygulamayı çekirdek güncellemeleri ile birlikte güncellemek için gerekli komutu girelim.\n$ sudo modprobe wireguard Aşağıda verilen komut girildiğinde beklenen sonuç.\n$ lsmod | grep wireguard wireguard 217088 0 ip6_udp_tunnel 16384 1 wireguard udp_tunnel 16384 1 wireguard Anahtarları üretelim\n$ cd /etc/wireguard $ umask 077 $ wg genkey | sudo tee privatekey | wg pubkey | sudo tee publickey VPN Konfigurasyon dosyasını /etc/wireguard/wg0.conf ayarlayalım.\n[Interface] PrivateKey = \u0026lt;daha-öncesinde-üretilen-gizli-anahtar\u0026gt; Address = 10.120.120.2/24 Address = fd86:ea04:1111::1/64 SaveConfig = true PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o ens3 -j MASQUERADE PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o ens3 -j MASQUERADE ListenPort = 51820 Burada önemli nokta ens3, ip tables komutu içerisinde yer alan en3, sunucudan sunucuya farklılık gösterebilir, bundan dolayı sizin sunucunuzda ne ise ağ kartının ismi onu girmelisiniz. ifconfig komutu sayesinde öğrenilebilir.\nBir diğer önemli nokta ise daha öncesinde 4. adımda üretilen privatekey, içeriğinin PrivateKey alanına girilmesidir.\nAğ trafiğini yönlendirme\n/etc/sysctl.conf dosyası içerisine aşağıda verilen bilgileri girerek kaydediniz.\nnet.ipv4.ip_forward=1 net.ipv6.conf.all.forwarding=1 Bilgiler gerekli dosyaya kaydedildikten sonra aşağıdaki komutlar sırası ile girilmelidir.\n$ sysctl -p $ wg-quick up wg0 Komutların girilmesi ve herhangi bir sorun görülmemesi durumda ve wg komutu terminale girildikten sonra aşağıda verilen çıktıya benzer bir çıktı göreceksiniz.\n$ wg interface: wg0 public key: loZviZQpT5Sy4gFKEbk6Vc/rcJ3bH84L7TUj4qMB918= private key: (hidden) listening port: 51820 Eğer herhangi bir sorun ile karşılaşmazsanız bu adıma kadar, bu demek oluyor ki, sunucu tarafında işiniz şimdilik tamamlandı geriye sadece kendi bilgisayarımızı, telefonumuzu vs VPN sunucusuna bağlamak kaldı.\nKullanıcı Ayarları Kullanıcıların kendi bilgisayar ortamlarında, telefonlarında, tabletlerinde veya diğer sunucularında kullanabileceği uygulamaları buradan indirebilirsiniz.\nGerekli uygulamayı kendi ortamınıza indirdikten sonra tek yapmanız gereken, VPN sunucusu tarafında ayarladığımız VPN\u0026rsquo;e bağlanmak, bunun için gerekli olan sadece konfigurasyonları doğru girmek olacaktır.\nKullanıcı tarafında, uygulama üzerinden aşağıda verilen konfigurasyona benzer bir ayarı (kendi kurduğunuz VPN ayarlarına göre privatekey ve ip adressi değişiklik gösterecektir.) ayarlamanız gerekmektedir.\n[Interface] Address = 10.120.120.2/32 Address = fd86:ea04:1111::2/128 # note that privatekey value is just a place holder PrivateKey = KIaLGPDJo6C1g891+swzfy4LkwQofR2q82pFR6BW9VM= DNS = 1.1.1.1 [Peer] PublicKey = \u0026lt;sunucunuza-ait-public-anahtar\u0026gt; Endpoint = \u0026lt;sunucunuzun-dış-ip-adresi\u0026gt;:51820 AllowedIPs = 0.0.0.0/0, ::/0 Gerekli işlemler kullanıcı tarafında da sağlandıktan sonra, sunucu tarafında bu kullanıcıya bağlantı izni vermek kalıyor, onuda aşağıda verilen komut ile sağlayabilirsiniz.\n$ wg set wg0 peer \u0026lt;kullanici-public-anahtari\u0026gt; allowed-ips 10.120.120.2/32,fd86:ea04:1111::2/128 Sunucu tarafindan kullanicinin VPN baglantisi sağladığını aşağıda verilen komut ile teyit edebilirsiniz.\n$ wg interface: wg0 public key: loZviZQpT5Sy4gFKEbk6Vc/rcJ3bH84L7TUj4qMB918= private key: (hidden) listening port: 51820 peer: Ta9esbl7yvQJA/rMt5NqS25I/oeuTKbFHJu7oV5dbA4= allowed ips: 10.120.120.2/32, fd86:ea04:1111::2/128 Daha sonrasinda, wireguard tarafından oluşturulan ağ kartını aktivate edelim.\n$ wg-quick up wg0 Güvenlik Duvarı ayarları Bazen sunucu tarafında yapmanız gereken bazı güvenlik duvarı ayarları bulunmakta, bunlar VPN bağlantısını başarılı bir şekilde sağlamanız için kritik öneme sahiptir.\n$ ufw enable VPN uygulamasına bağlanmamızı sağlayacak portu açıyoruz.\n$ ufw allow 51820/udp IP tabloları ile 51820 portu için bazı ayarlamalar yapıyoruz.\n$ iptables -A INPUT -p udp -m udp --dport 51820 -j ACCEPT $ iptables -A OUTPUT -p udp -m udp --sport 51820 -j ACCEPT Burada önemli olan kısımlardan biriside bütün komutlar ROOT, yani yönetici yetkisi ile yapılmalı, aksi takdirde hata verecektir.\nBu noktadan sonra, bilgisayarınıza, tabletinize veya telefonunuza kurduğunuz WireGuard uygulaması sayesinde sorunsuz ve güvenlikli bir şekilde internetinizi kullanabilirsiniz.\n","permalink":"https://mrturkmen.com/posts/vpn-kuralim/","summary":"\u003ch1 id=\"vpn-kuralım\"\u003eVPN Kuralım\u003c/h1\u003e\n\u003cp\u003eBugün sizlere kendinize ait VPN sistemi nasıl kurulur, onu anlatmak istiyorum, daha önce İngilizce olarak, \u003ca href=\"https://mrturkmen.com/setup-free-vpn/\"\u003eyayınladım\u003c/a\u003e fakat  Türkçe bir kaynağın da faydalı olabileceğini düşündüm. Burada anlatılanlar, ubuntu ailesine (16.04,18.04) ait sunucular üzerinde test edilmiştir.\u003c/p\u003e\n\u003cp\u003eİlk olarak bulut hizmeti sağlayan bir şirketten bu DigitalOcean, Google Cloud, Microsoft Azure veya Amazon olabilir, sunucu kiralıyorsunuz, en ucuzu ve makul olanı DigitalOcean tarafından sunulan \u003ca href=\"https://www.digitalocean.com/pricing/\"\u003eaylık 5 dolar\u003c/a\u003e olan sunucu diyebilirim. Sunucuyu kiraladıktan ve ssh bağlantısını sağladıktan sonra VPN kurulumuna geçebiliriz.\u003c/p\u003e","title":"wireguard: kendimize özel vpn kurulumu "},{"content":"Concurrency in Go Concurrency in Go, makes Go programming language very unique and attractive compared to other languages, in this section I am going to share the notes which I took when I was watching coursera video series.\nIf you did not already check previous post on Go, it could be helpful to check it out first.\nGo Notes (OOP) Specialization serie is Programming with Google Go Keep in mind that the notes are taken from several resources mainly from the course, however post may include other resources as well, I have referenced them when required, if nothing is referenced then it means, notes are taken from the course.\nParalel Execution Two programs execute in paralel if they execute at exactly the same time At time t, an instruction is being performed for both P1 and P2. Why use parallel execution Tasks may complete more quickly Example: Two Piles of dishes to wash Two dishwashers can complete twice as fast as one Some tasks must be performed sequentially Example: Wash dish, dry dish Must wash before you can dry Some tasks are parallelizable and some are not Von Neumann Bottleneck Speedup without Parallelism Can we achieve speedup without Parallelism ? Design faster processors Get speedup without changing software Design processor with more memory Reduces the Von Heumann bottleneck Cache access time=1 clock cycle Main memory access time = ~100 clock cycles Increasing on-chip cache improves performance Moore\u0026rsquo;s Law Predicted that transistor density would double every two years Smaller transistors switch faster Not a physical law, just an observation Exponential increase in density would lead to exponential increase in speed Power Wall Power / Temperature Problem Transistors consume power when they switch Increasing transistor density leads to increased power consumption Smaller transistors use less power, but density scaling is much faster High power leads to high temperature Air cooling (fans) can only remove so much heat Dynamic Power P = a * CFV^2 a is percent of time switching C is capacitance (related to size) F is the clock frequency V is voltage swing (from low to high) Voltage is important 0 to 5V uses much more power then 0 to 1.3V Dennard Scaling Voltage should scale with transistor size Keeps power consumption and temperature, low Problem: Voltage can\u0026rsquo;t go too low Must stay above threshold voltage Noise problems occur Problem: Does not consider leakage power Dennard scaling must stop Multi-Core Systems P = a * CFV^2 Cannot increase frequency Can still add processor cores, without increasing frequency Trend is apparent today Parallel execution is needed to exploit multi-core systems Code made to execute on multiple cores Different programs on different cores Concurrent vs Parallel Concurrent Execution Concurrent execution is not necessarily the some as parallel execution Concurrent: start and end times overlap Parallel: execute at exactly the same time Parallel tasks must be executed on different hardware Concurrent tasks may be executed on the same hardware Only one task actually executed at a time Mapping from tasks to hardware is not directly controlled by programmer At least not in go Concurrent Programming Programmer determines which tasks can be executed in parallel Mapping tasks to hardware Operating system Go runtime scheduler Hiding Latency *(Crucial) Concurrency improves performance even without parallelism\nTasks must periodically wait for something\ni.e. wait for memory X = Y+Z read Y,Z from memory May wait 100+ clock cycles Other concurrent tasks can operate while one task is waiting\nConcurrent programming is useful even no paralelism\nHardware Mapping Programmer does not determine the hardware mapping Programmer makes parallelism possible Hardware mapping depends on many factors Where is the data ? What are the communication costs ? Concurrency Basics Processes An instance of a running program Things unique to a process Memory a. Virtual address space b. Code, stack, heap, shared libraries Registers a. Program counter Operating System Allows many processes to execute concurrently Processes are switched quickly 20ms User has the impression of parallelism Operating system must give processes fair access to resources Scheduling Processes Operating system schedules processes for execution Gives the illusion of parallel execution OS gives fair access to CPU, memory , etc. Context Switch Control flow changes from one process to another Process \u0026ldquo;context\u0026rdquo; must be swapped Air cooling (fans) can only remove so much heat Threads and Goroutines Threads vs. Processes Threads share some context Many threads can exist in one process OS schedules threads rather than processes Goroutines Like a thread in Go Many Goroutines execute within a single OS thread We can call Goroutines as lightweight threads in Go.\nGo Runtime Scheduler Schedules goroutines inside an OS thread Like a little OS inside a single OS thread Logical processor is mapped to a thread Interleavings Order of execution within a task is known Order of execution between concurrent tasks is unknown Interleaving of instructions between tasks is unknown Possible Interleavings Many interleavings are possible Must consider all possibilities Ordering is non-deterministic Interleavings occurs on machine code level. Race Conditions It is happenning when multiple goroutines try to write/read from a source at the same time, could be prevented using mutex\nOutcome depends on non-deterministic ordering Races occur due to communication Communication Between Tasks Threads are largely independent but not completely independent Web server, one thread per client. Make threads for each client who are connecting to web server Image processing, 1 thread per pixel block Some level of communication can occur between threads, let\u0026rsquo;s say we want to blur the image accordingly, then the threads which are working on neighborhood pixels should communicate with each other. Goroutines Goroutines are new way of threading in lightweight approach.\nCreating a Goroutine One goroutine is created automatically to execute the main() Other goroutines are created using the go keyword package main func main () { a=1\tfoo()\ta=2 } func foo() { // does something ... } Main goroutine blocks on call to foo() package main func main () { a=1\tgo foo()\ta=2 } func foo() { // does something ... } New goroutine created for foo() Main goroutine does not block Exiting a Goroutine A goroutine exits when its code is complete When the main goroutine is complete, all other goroutines exit A goroutine may not complete its execution because main completes early Early Exit func main() { go fmt.Printf(\u0026#34;New routine\u0026#34;) fmt.Printf(\u0026#34;Main routine\u0026#34;) } Only \u0026ldquo;Main routine\u0026rdquo; is printed Main finished before the new goroutine started Delayed Exit func main() { go fmt.Printf(\u0026#34;New routine\u0026#34;) time.Sleep(100 * time.Milisecond) fmt.Printf(\u0026#34;Main routine\u0026#34;) } Add a delay in the main routine to give the new routine a chance to complete \u0026ldquo;New RouteMainRoutine \u0026quot; is now printed Timing with Goroutines Adding a delay to wait for a goroutine is bad !! Timing assumptions may be wrong Assumption: delay of 100 ms will ensure that goroutine has time to execute Maybe the OS schedules another thread Maybe the Go runtime schedules another goroutine Timing is nondeterministic Need formal synchronization constructs Synchronization Using global events whose execution is viewed by all threads, simultaneously Want print to occur after update of x Example\nTask 1 Task 2 x=1 x=x+1 GB if GB print x GLOBAL EVENT (GB) is viewed by all tasks at the same time Print must occur after update of x Synchronization is used to restrict bad interleavings Threads in Go Threads in Go are generally handled using wait groups\nSync WaitGroup Sync package contains functions to synchronize between goroutines sync.WaitGroup forces a goroutine to wait for other goroutines Contains an internal counter Increment counter for each goroutine to wait for Decrement counter when each goroutine Waiting goroutine cannot continue until counter is 0 Using WaitGroup Add() increments the counter Done() decrements the counter Wait() blocks until counter == 0 Example func foo(wg *sync.WaitGroup) { fmt.Printf(\u0026#34;New routine\u0026#34;) wg.Done() } func main() { var wg sync.WaitGroup wg.Add(1) go foo(\u0026amp;wg) wg.Wait() fmt.Printf(\u0026#34;Main Routine\u0026#34;) } Goroutine Communication Goroutines usually work together to perform a bigger task Often need to send data to collaborate Example : Find the product of 4 integers Make 2 goroutines, each multiplines a pair Main goroutine multiplies the 2 results Need to send ints from main routine to the two sub-routines Need to send results from sub-routines back to main routine Channels Channels are used to make communication between goroutines Channels are typed Use make() to create a channel c:=make(chan int) Send and receive data using the \u0026lt;- operator Send data on a channel C \u0026lt; - 3 Receive data from a channel x:= \u0026lt;- c Example func prod(v1 int, v2 int, c chan int) { c \u0026lt;- v1*v2 } func main() { c:=make(chan int) go prod(1,2,c) go prod(3,4,c) a:= \u0026lt;-c b:= \u0026lt;-c fmt.Println(a*b) } Unbuffered Channel Unbuffered channels cannot hold data in transit Default is unbuffered Sending blocks until data is received Receiving blocks until data is sent Blocking and Synchronization Channel communication is synchronous Blocking is the same as waiting for communication Receiving and ignoring the result is same as a Wait() Buffered Channel Channel Capacity Channel can contain a limited number of objects Default size 0 (unbuffered) Capacity is the number of objects, it can hold in transit Optional argument to make() defines channel capacity c:=make(chan int, 3) Sending only blocks if buffer is full Receiving only blocks if buffer is empty Channel Blocking, Receive Channel with capacity 1 First receive blocks until send occurs Second receive blocks forever Channel Blocking, Send Second send blocks until receive is done Receive can block until first send is done Use of Buffering Sender and receiver do not need to operate at exactly the same speed Adding a channel to our goroutine Note that in this section notes are taken from: https://www.sohamkamani.com/blog/2017/08/24/golang-channels-explained\nA channel gives us a way to \u0026ldquo;connect \u0026quot; the different concurrent parts of our program. Channels can be thought of as \u0026ldquo;pipes\u0026rdquo; or \u0026ldquo;arteries\u0026rdquo; that connect the different concurrent parts of our code Directionality out chan \u0026lt;- int ` The chan\u0026lt;- declaration tells us that you can only put stuff into the channel, but not receive anything from it. The int declaration tells us that the \u0026ldquo;stuff\u0026rdquo; you put into the channel can only be of the int datatype Although they look like seperate parts, chan\u0026lt;-int can be thought of as one datatype, that describes a \u0026ldquo;send-only\u0026rdquo; channel of integers. Similarly, an example of a \u0026ldquo;receive-only\u0026rdquo; channel declaration would look like: out \u0026lt;- chan int A channel can be declared without giving any directionatility, which means it can send or receive data.\nBi-directional channel can be created using following \u0026ldquo;make\u0026rdquo; statement\nout :=make(chan int) After this section, notes are taken from: https://go101.org/article/channel.html if you prefer to have deep dive into channels, you may visit there.\nConcurrent computations may share resources, generally memory resource. There are some circumstances may happen in a concurrent computing.\nIn the same period of one computation is writing data to a memory segment, another computation is reading data from the same memory segment. Then the integrity of the data read by the other computation might be not preserved.\nIn the same period of one computation is writing data to a memory segment, another computation is also writing data to the same memory segment. Then the integrity of the data stored at the memory segment might be not preserved.\nThese circumstances are called data races. One of the duties in concurrent programming is to control resource sharing among concurrent applications, so that data races will NOT happen. The ways to achieve this duty are called concurrency synchronization, or data synchronization. GO supports several data synchronization techniques. The following section will introduce one of them, channel.\nOther duties in concurrent programming include:\nDetermine how many computations are needed Determine when to start, block, unblock and end a computation. Determine how to distribute workload among concurrent computations. Most operations in Go are not synchronized. In other words, they are not concurrency-safe. These operations include value assignments, argument passing and container element manipulations. There are only a few operations which are synchronized, including the several to be introduced channel operations below.\nDO NOT COMMUNICATE BY SHARING MEMORY, SHARE MEMORY BY COMMUNICATING *(THROUGH CHANNELS)\nChannel Types and Values Like array, slice and map, each channel type has an element type. A channel can only transfer values of the element type of (the type of) the channel.\nChannel types can be bidirectional or single-directional. Assume T is an arbitrary type,\nchan T denotes a bidirectional channel type. Compilers allow both receiving values from and sending values to bidirectional channels. chan \u0026lt;-T denotes a send-only channel type. Compilers do not allow receiving values from send-only channels. \u0026lt;- chan T denotes a receive-only channel type. Compilers do not allow sending values to receive-only channels. T is called element types of these channel types.\nValues of bidirectional channel type chan T can be implicitly converted to both send-only type chan \u0026lt;-T and receive-only type \u0026lt;-chan T , but not vice versa (even if explicitly). Values of send only type chan\u0026lt;-T cannot be converted to receive only type \u0026lt;-chan T.\nEach channel has capacity. A channel value with a zero capacity is called unbuffered channel and a channel value with a non-zero capacity is called buffered channel.\nFor more detailed explanation on channels you can visit https://go101.org/article/channel.html\nTake care ! 👋🏻\nMaybe next time, topics could be revisited by examples 😉\n","permalink":"https://mrturkmen.com/posts/go-concur/","summary":"\u003ch1 id=\"concurrency-in-go\"\u003eConcurrency in Go\u003c/h1\u003e\n\u003cp\u003eConcurrency in Go, makes Go programming language very unique and attractive compared to other languages, in this section I am going to share the notes which I took when I was watching coursera video series.\u003c/p\u003e\n\u003cp\u003eIf you did not already check previous post on Go, it could be helpful to check it out first.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/go-notes/\"\u003eGo Notes (OOP)\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSpecialization serie is \u003ca href=\"https://www.coursera.org/specializations/google-golang\"\u003eProgramming with Google Go\u003c/a\u003e\n\u003c!-- raw HTML omitted --\u003e\u003c/p\u003e","title":"go: concurrency notes "},{"content":"In this post, I would like to share the notes that I took when I was completing following Go series education on Coursera. I can recommend it for anyone who would like to get rapid introduction to Go programming.\nProgramming with Google Go\nI should admit that although it looks fancy, nothing can be compared to actual development and contribution to open source projects. The course itself is quite interesting and contains very handy exercises regarding to Go development mentality. I would recommend it for anyone who has some interest in Go development and do not know where to start.\nThere will be following headers and subheaders which contains some notes on Go Programming language as bulletpoints.\nOOP in Go In Go, there is no such a concept of object oriented programming as in Java, however, it is possible to imply similar approach (object oriented approach) in Go using interfaces and structs.\nClasses There is no \u0026ldquo;Class\u0026rdquo; keyword in Go.\nCollection of data fields and functions that share a well-defined responsibility Example: Point class Used in a geometry program Data: x coordinate, y coordinate Funtions: DistToOrigin(), Quadrant() AddXOffSet(), AddYOffset() Classes are template Contain data fields, not data Classes are supported with structs in Go.\ntype Point struct { X float64 Y float64 } Structs with methods, structs and methods together allow arbitrary data and functions to be composed. func (p Point) DistToOrig() { t:= math.Pow(p.x,2) + math.Pow(p.y,2) return math.Sqrt(t) } func main() { p1 := Point(3,4) fmt.Println(p1.DistToOrig()) } Objects Instance of a class Contains real data Example: Point Class Encapsulation Data can be protected from the programmer Data can be accessed only using methods Maybe we do not trust the programmer to keep data consistent Example: Double distance to origin Option 1: Make method DoubleDist Option 2: Trust programmer to double X and Y directly Encapsulation is supported as following ;\nCreate a package called data which has exported function which is PrintX, since it starts with capital letter, in Go, when something starts with capital letter, it means that it is exported\npackage data var x int=1 func PrintX() { fmt.Println(x) } Main package which is starting point of any application in Go.\npackage main func main() { data.PrintX() } Controlling access to Structs\nHide fields of structs by starting field name with a lower-case letter. Define public methods which access hidden data package data type Point struct{ x float64 y float64 } func (p *Point) InitMe(xn,xy float64) { p.x =xn p.y =xy } func (p *Point) Scale(v float64) { p.x=p.x*v p.y=p.y*v } func (p *Point) PrintMe() { fmt.Println(p.x,p.y) } Note that all methods are public ! Since their initial character is capital letter.\npackage main func main() { var p data.Point p.InitMe(3,4) p.Scale(2) p.PrintMe() } Access to hidden fields can only be possible through public access. Limitation of Method Receiver is passed implicitly as an argument to the method\nMethod cannot modify the data inside the receiver\nExample: OffsetX() should increase x coordinate\npackage main func main(){ p1:=Point(3,4) p1.OffsetX(5) } Large Receivers\nIf receiver is large, lots of copying is required type Image [100] [100]int func main() { i1 := GrabImage() i1.BlurImage() } 10.000 ints copied to BlurImage() (Pitfalls)\nPointer Receivers\nfunc (p *Point) OffsetX (v float64) { p.x=p.x+v } Receiver can be a pointer to a type Call by reference, pointer is passed to the method Point Receivers, Referenceing, Dereferencing No need to dereference func (p *Point) OffsetX (v int) { p.x=p.x+v } Point is referenced as p, not *p\nNo need to reference\npackage main func main() { p:=Point{3,4} p.OffsetX(5) fmt.Println(p.x) } Do not need to reference when calling the method Good Programming Practices\nAll methods for type have pointer receivers or All methods for a type have non-pointer receivers Mixing pointer/non-pointer receivers for a type will get confusing ! Pointer receiver allows modification Polymorphism Ability for an object to have different forms depending on the context Example Area() function Rectangle area is base * height Triangle area is 0.5*base*height Identical at a high level of abstraction Different at a low level of abstraction Inheritance (No Inheritance in GoLang) Sublcass inherits the methods/data of the superclass\nExample: Speaker superclass\nSpeak() method, pring \u0026quot;\u0026lt;noise\u0026gt; \u0026quot; Subclassses Cat and Dog\nAlso have the Speak() method Cat and Dog are different forms of speaker\nRemember: Go does not have inheritance\nOverriding Subclass redefines a method inherited from the superclass\nExample: Speaker, Cat, Dog\nSpekaer Speak() prints \u0026ldquo;\u0026rdquo; Cat Speak() prints \u0026ldquo;meow\u0026rdquo; Dog Speak() prints \u0026ldquo;woof\u0026rdquo; Speak() is polymorphic\nDifferent implementations for each class Same signature (name, params and return) Interface Set of method signatures\nName, parameters, return values Implementation is NOT defined Used to express conceptial similarity between types.\nExample : Shape2D interface\nAll 2D shapes must have Area() and Perimeter()\nSatisfying an Interface Type satisfies an interface if type defines all methods specified in the interface. Same method signatures Rectangle and Triangle types satisfy the Shape2D interface Must have Area() and Perimeter() methods Additional methods are OK. Similar to inheritance with overriding. Example type Shape2D interface { Area() float64 Perimeter() float64 } type Triangle {...} func (t Triangle) Area() float64 {....} func (t Triangle) Perimeter() float64 {....} Triangle type satisfies the Shape2D interface No need to state it explicitly Interface vs Concrete Types Concrete Types Specify the exact representation of the data and methods Complete method implementation is included Interface Types Specifies some method signatures Implementations are abstracted Interface Values Can be treated like other values Assigned to variables Passed, returned Interface values have two components Dynamic Type : Concrete type which it is assigned to Dynamic Value: Value of the dynamic type Defining an interface type type Speaker interface { Speak()} type Dog struct {name string } func (d Dog) Speak() { fmt.Println(d.name) } func main() { var s1 Speaker var d1 Dog{\u0026#34;Brian\u0026#34;} s1=d1 s1.Speak() Dynamic type is Dog and dynamic value is d1.\nAn interface can have a nil dynamic value\nvar s1 Speaker var d1 *Dog s1=d1 d1 has no concrete value yet s1 has a dynamic type but no dynamic value Nil Dynamic Value Can still call the Speak() method of s1 Does not need a dynamic value to call Need to check inside the method func (d *Dog)Speak() { if d==nil{ fmt.Println(\u0026#34;\u0026lt;noise\u0026gt;\u0026#34;) }else{ fmt.Println(d.name) }\t} var s1 Speaker var d1 *Dog s1=d1 s1.Speak() // it works, since s1 is mapped to d1 Nil Interface Value Interface with nil dynamic type Very different from an interface with a nill dynamic value Nil dynamic value and valid dynamic type\nvar s1 Speaker var d1 *Dog s1=d1 Can call a method since type is known, Nil dynamic type var s1 Speaker *(there is no actually method to call) Cannot call a method, runtime error No dynamic type and no dynamic value then you cannot call the interface…\nUsing Interfaces Need a function which takes multiple types of parameter Function foo() parameter Type X or Type Y Define interface Z foo() parameter is interface Z Types X and Y satisfy Z Interface methods must be those needed by foo() Example Interface for Shapes Pool in a Yard\nI need to put a pool in my yard Pool need to fit in my yard Total area must be limited Pool needs to be fenced Total perimeters must be limited Need to determine if a pool shape satisfies criteria FitInYard() Takes a shape as argument Returns true if the shape satisfies criteria FitInYard() Many Possible shape types Rectangle, triangle, circle FitInYards() should take many shape types Valid shape types must have Area() Perimeter() Any shape with these methods is OK. type Shape2D interface { Area() float64 Perimeter() float64 } type Triangle {...} func (t Triangle) Area() float64 {...} func (t Triangle) Perimeter() float64 {...} type Rectangle {...} func(t Rectangle) Area() float64 {...} func (t Rectangle) Perimeter() float64 {....} Rectangle and Triangle satisfy Shape2D interface.\nFitInYard() Implementation\nfunc FitInYard(s Shape2D) bool { if (s.Area() \u0026gt; 100 \u0026amp;\u0026amp; s.Perimeter() \u0026gt; 100) { return true } return false } Empty Interface Empty interface specifies no methods All types satisfy the empty interface Use it to have a function accept any type as a parameter func PrintMe(val interface{} ) { fmt.Println(val) } Type Assertions Concealing Type Differences__ Interfaces hide the differences between types func fitInYard(s Shape2D)bool { if (s.Area() \u0026gt;100 \u0026amp;\u0026amp; s.Perimeter()\u0026gt;100){ return true } return false } Sometimes you need to treat different types in different ways Exposing Type Differences Example: Graphics program\nDrawShape() will draw any shape\nfunc DrawShape (s Shape2D) {..... } Underlying API has different drawing functions for each shape\nfunc DrawRect (r Rectangle) {.... func DrawTriangle(t Triangle) {... Concrete type of shape s must be determined\nType Assertions for Disambiguation Type assertions can be used to determine and extract the underlying concrete type func DrawShape(s Shape2D) bool { rect,ok :=s.(Rectangle) if ok { DrawRect(rect) } tri,ok := s.(Triangle) if ok { DrawRect(tri) } } Type assertion extracts Rectangle from Shape2D Concrete type in parentheses If interface contains concrete type rect == concrete type, ok == true If interface does not contain concrete type rect==zero, ok==false Type Switch Switch statement used with a type assertion func DrawShape(s Shape2D) bool { switch:= sh:=s.(type) { case Rectangle: DrawRect(sh) case Triangle: DrawTri(sh) } } Error Interface Many Go programs return error interface objects to indicate errors type error interface { Error() string } Correct operation : error==nil Incorrect operation: Error() print error message Handling Errors f,err := os.Open(\u0026#34;/harris/text.txt\u0026#34;) if err!=nil { fmt.Println(err) return } Check whether the error is nil If it is not nil, handle it fmt package calls the error() method to generate string to print Keep in mind that the topics which are mentioned on this post is just brief summary, it means that all subheaders and headers can be extended to any size, however these are just bulletpoints and overall information in Functions, Methods, and Interfaces in Go module of specialization serie.\nIn next post, notes which are taken from concurrency module of the specialization serie will be posted.\nTake care ! 👋🏻\n","permalink":"https://mrturkmen.com/posts/go-notes/","summary":"\u003cp\u003eIn this post, I would like to share the notes that I took when I was completing following Go series education on Coursera. I can recommend it for anyone who would like to get rapid introduction to Go programming.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://www.coursera.org/specializations/google-golang\"\u003eProgramming with Google Go\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Go course certificate\" loading=\"lazy\" src=\"../../images/go_certificate.png\"\u003e\u003c/p\u003e\n\u003cp\u003eI should admit that although it looks fancy, nothing can be compared to actual development and contribution to open source projects.\nThe course itself is quite interesting and contains very handy exercises regarding to Go development mentality. I would recommend it for anyone who has some interest in Go development and do not know where to start.\u003c/p\u003e","title":"go: object oriented programming notes "},{"content":"Why VPN? It is crucial to do not expose your personal details or not being attacked by someone when you are connected to public endpoints such as coffee, airport, hotel and guest WIFI points. Furthermore, sometimes, people require to have organizatinal VPN access if organization itself does not provide one. For instance when students in universities have taken into consideration, it is quite important to reach resources that university is providing, it could be IEEE library access or enclosed resource which is only exclusive to internal network of the organization.\nNo matter what is your intention to use VPN, it is quite useful to have in any case.\nThere are lots of tutorials and explanations regarding to setting up WireGuard however I could not manage to find proper instructions for MacOSX users that\u0026rsquo;s why I thought it could be useful to pack up in once in a post. I hope, this would be handy for anyone who are seeking for brand new tutorial for using WireGuard on MacOSx.\nPreliminary information; there is no such a concept of server and client in WireGuard, all devices are called as peers. However, in sake of understanding better, I will refer virtual private server as WireGuard server however, it is actually a PEER !\nPrerequirement Wireguard is a decent VPN solution with all recent crypto features compared to other open source VPN solutions furthermore, it is lightweight.\nWireguard works based on key-pair relation between server and client just like ssh connection. The setup for personal usage is quite simple to do. Moreover, it is super fast, simple and more performant than any other open source VPN solution. Let\u0026rsquo;s start build our own VPN solution using WireGuard.\nPrepare the environment to setup WireGuard Following scenario is made on ubuntu 18.04, however almost for any linux distribution the steps are more or less same.\nVirtual private server (VPS), cheapest VPS with ubuntu 18.04 could be enough to get your own personal VPN. It can be rented over ;\nGoogle Cloud: Google provides $300 for 12 months for new comers to Google Cloud. Amazon AWS: Amazon has some free tiers which could be suitable for running your own VPN solution. Check it out from their website. Digital Ocean: Provides cheapest VPS on demand like $5 per month, which could be enough for personal VPN. Furthermore, you can claim $50 for Digital Ocean from Github Students Package, if you have student account on Github. Microsoft Azure: If you have Github student account, you can take some advantages of Microsoft cloud services as well. Check it out from Github Student Pack Setup Peer 1 [-Peer 1 is the Server, which will route all or some of your network traffic- ] I assumed that you have bought your VPS and ready to go for installation steps for WireGuard. Make sure that you have done with your SSH connection.\n$ sudo apt-get update \u0026amp;\u0026amp; sudo apt-get upgrade -y $ sudo add-apt-repository ppa:wireguard/wireguard $ sudo apt-get update $ sudo apt-get install wireguard Syncing Wireguard with kernel updates, which means that whenever there is an update for your linux kernel, wireguard module will be updated too.\n$ sudo modprobe wireguard Afterwards, you can ensure that the module is loaded as following;\n$ lsmod | grep wireguard wireguard 217088 0 ip6_udp_tunnel 16384 1 wireguard udp_tunnel 16384 1 wireguard Generate Keys Keys are backbone of WireGuard, extremely important step to take into.\n$ cd /etc/wireguard $ umask 077 $ wg genkey | sudo tee privatekey | wg pubkey | sudo tee publickey After given commands above, you will have publickey and privatekey in /etc/wireguard/ directory.\nSetup Configuration In this step, an interface should be defined in order to route all traffic from clients over rented VPS. Common interface for WireGuard is wg0.\nCreate a configuration file on VPS as /etc/wireguard/wg0.conf through your favorite text editor (vim, nano, vi).\n[Interface] PrivateKey = \u0026lt;generated-private-key-here\u0026gt; Address = 10.120.120.2/24 Address = fd86:ea04:1111::1/64 SaveConfig = true PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o ens3 -j MASQUERADE PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o ens3 -j MASQUERADE ListenPort = 51820 ens3 is main network interface fo your server, if it is different change it with correct one, you can observe your by typing ifconfig.\nPrivateKey is the key which was generated in generate keys step.\nForward Traffic Traffic which comes into VPS should be forwarded properly, in order to set it up, change /etc/sysctl.conf file as follows.\nUsing your text editor, add following two lines of configuration into /etc/sysctl.conf\nnet.ipv4.ip_forward=1 net.ipv6.conf.all.forwarding=1 After that, to activate them immeditaly, type;\n$ sysctl -p It is time to bring VPN interface up;\n$ wg-quick up wg0 If there is no problem or complain when the command above called, you should be able to get following results from given command below.\n$ wg interface: wg0 public key: loZviZQpT5Sy4gFKEbk6Vc/rcJ3bH84L7TUj4qMB918= private key: (hidden) listening port: 51820 Setup Peer 2 (-client-) In my case, I will gonna go over for setup for MacOS client, however, the logic remains same for any other client that you will use.\nAll required appliations are downloadable from given adress below.\nApplications for all distribution After installation of your application from given official website (https://www.wireguard.com/install/), it is time to add peers to server, and configure Peer 2 (client).\nSetup Configuration (client) If you are using MacOsX, you need to add following configuration under /usr/local/etc/wireguard/wg0.conf, however if you are using Linux, /etc/wireguard/wg0.conf would be correct place to put the configuration given below.\n[Interface] Address = 10.120.120.2/32 Address = fd86:ea04:1111::2/128 # note that privatekey value is just a place holder PrivateKey = KIaLGPDJo6C1g891+swzfy4LkwQofR2q82pFR6BW9VM= DNS = 1.1.1.1 [Peer] PublicKey = \u0026lt;your server public key\u0026gt; Endpoint = \u0026lt;your server public ip\u0026gt;:51820 AllowedIPs = 0.0.0.0/0, ::/0 If you have already installed GUI application from official installation website of WireGuard (https://www.wireguard.com/install/), it could be nice to arrange that configuration file as shown below.\nYou will be able to create your configuration file through WireGuard GUI application as follows.\nNote that the PrivateKey value is just a placeholder, it is not valid value to put into, you need to take care of your own private key which is generated by WireGuard.\nPublicKey : This should be same with the public key when you have generated the keys on server in this step PrivateKey: This is the key which is generated by WireGuard when you start to create new ocnfiguration tunnel from the applicaiton that you have installed from official site. EndPoint: IP address or A record of your VPS. Add Peer When you are done with setting up configuration in client side, you need to let your other peer (server) know your device. It can be achieved by following command on server.\n$ wg set wg0 peer \u0026lt;client-public-key\u0026gt; allowed-ips 10.120.120.2/32,fd86:ea04:1111::2/128 client-public-key: It is a placeholder, change it with your own public key which can be seen in your WireGuard application, e.g; it is shown in given figure above. When everything goes fine, you will be able to list peers by executing following command on server side.\n$ wg interface: wg0 public key: loZviZQpT5Sy4gFKEbk6Vc/rcJ3bH84L7TUj4qMB918= private key: (hidden) listening port: 51820 peer: Ta9esbl7yvQJA/rMt5NqS25I/oeuTKbFHJu7oV5dbA4= allowed ips: 10.120.120.2/32, fd86:ea04:1111::2/128 Last but not least step is to up wg0 interface on client side.(peer 2)\n$ wg-quick up wg0 Now you can check your IP address location through https://whatismyipaddress.com/\nFirewall configuration It can be handy to enable ufw and allow port 51820/udp for traffic routing.\nCheck ufw status $ ufw status verbose If not enabled, enable it by ; (before enabling ufw make sure that you have already allowed port 22 otherwise you may face some problems in return) $ ufw enable Allow port 51820/udp $ ufw allow 51820/udp Sometimes it is required to setup iptables, if that is the case use following rule.\n$ iptables -A INPUT -p udp -m udp --dport 51820 -j ACCEPT $ iptables -A OUTPUT -p udp -m udp --sport 51820 -j ACCEPT Note: Run commands with a user which has root privileges\nHow Traffic Looks Like Following figures are just snippet of some network traffic which is captured by WireShark. In the first figure, WireGuard is disabled, however in second image, WireGuard is enabled, you can easily distinguish the difference.\nIn ordinary usage (without VPN), your network traffic is seen as follows;\nWhen you have configured your WireGuard to route all your traffic, here is how it is look like;\nIt can be observed that all the traffic which goes out from client side has been encrypted with WireGuard.\nAll in all, it is quite handy to have your own private VPN access which decreases the likelihood of being attacked. Apart from VPN capabilities of WireGuard, it stabilize WIFI connections and actually improves quality of WiFi connection on clients. Keep in mind that you need to create exclusive key pairs for each client who would like to route all or some of his/her traffic over VPN.\nReferences Quick Start Guide Installation Guide Unofficial WireGuard Docs Performance Limitation ","permalink":"https://mrturkmen.com/posts/setup-free-vpn/","summary":"\u003ch2 id=\"why-vpn\"\u003eWhy VPN?\u003c/h2\u003e\n\u003cp\u003eIt is crucial to do not expose your personal details or not being attacked by someone when you are connected to public endpoints such as coffee, airport, hotel and guest WIFI points. Furthermore,  sometimes, people require to have organizatinal VPN access if organization itself does not provide one. For instance when students in universities have taken into consideration, it is quite important to reach resources that university is providing, it could be IEEE library access or enclosed resource which is only exclusive to internal network of the organization.\u003c/p\u003e","title":"wireguard: set it up in few steps"},{"content":"Summary In this post, one of the well known (-open to discuss :)-) error of \u0026ldquo;No space left on device\u0026rdquo; which is caused due to Docker will be solved with different approaches.\nNote: \u0026ldquo;No space left on device\u0026rdquo; error can be caused due to any other reason than docker itself. Hence, it would be nice to make sure that the error is caused due to docker volumes.\nYou can check whether it is caused due to docker volumes or not by following steps over here\nNote that the explanations and examples may differ according to your environment, hence instead of taking these information as rules, applying them as a reference would be much better approach.\nThe system information for particular scenario described on post is given below:\nSystem information PRETTY_NAME: Debian GNU/Linux 9 (stretch) NAME: Debian GNU/Linux VERSION_ID: 9 VERSION: 9 (stretch) KERNEL_RELEASE: 4.9.0-6-amd64 Docker daemon version Engine:\nVersion: 19.03.8 API version: 1.40 (minimum version 1.12) Go version: go1.12.17 Git commit: afacb8b7f0 Built: Wed Mar 11 01:24:36 2020 OS/Arch: linux/amd64 Experimental: false Although, it is NOT fully required to have same or somehow closer version of OS and Docker daemon, it could be nice to know exactly which conditions the following fix can work.\nMost of the cases, the following information could be valid for all Debian based systems however it is not for sure.\nThe Problem Docker can be used for most of the cases although now Kubernetes or any other orchestration systems preffered to used. That kind of orchestration systems may not cause this problem, hence they mostly provide auto scaling and runs on cloud. However, when you have a server with all responsibility, you may face this exact problem.\nThe problem is installation of docker into any system with default settings may create head ache in the future. If you are planning to use docker, intensively, the main reason of that, docker is generally using\n/var/lib/docker place for docker volumes. In most scenarios, servers do not use root path for storing information, instead the appropriate approach for storing information on server is creating data volumes which has huge capacity and ability to extend or shrink time to time. The problem starts to show off itself when docker containers have been used for long period of time.\nEnsure about the problem It is quite handy to check whether the error of \u0026ldquo;No space left on device\u0026rdquo; caused by docker volumes. To do that following simple bash commands can be used.\n$ df -h /var/lib This will display free and used disk space for /var/lib path. Afterwards, you can see how is the difference between free and used spaces among usage percentage. An example is given below, it is taken from the system that I mentioned at the beginning in system information section.\n$ df -h /var/lib Filesystem Size Used Avail Use% Mounted on /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/mapper/data-data 9.1T 402G 8.2T 5% /data udev 252G 0 252G 0% /dev /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / proc 0 0 0 - /proc /dev/sda3 173G 162G 2.4G 99% / tmpfs 51G 275M 51G 1% /run /dev/sda3 173G 162G 2.4G 99% / /dev/nvme0n1 1.5T 775G 618G 56% /scratch /dev/sda3 173G 162G 2.4G 99% / sysfs 0 0 0 - /sys /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / /dev/sda3 173G 162G 2.4G 99% / As you can observe from the output above, I have plenty of spaces for volume /dev/nvme0n1 and /dev/mapper/data-data, however I was getting \u0026ldquo;No space left on the device\u0026rdquo; error, because root path became full due to docker volumes.\nIt can be quickly checked by the command below.\n$ df /var/lib/docker Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda3 180572828 166898524 4432040 98% / Temporary solution A temporary solution would be pruning docker volumes by running ;\n$ docker volume prune it will prune all volumes which are NOT in-use, if the volumes which are creating this bunch of data are in-use, prune command will not have any effect on the error. Although it works, it is a temporary solution, it may re-trigger the error in future. To resolve this issue permanently following approach could be used: Permanent Solution\nThe approach of having temporary solution can be extended with cronjobs, however, it is definitely not nice way of handling issues.\nHowever, it could be nice to have cron jobs which prune docker system time to time, because old docker images, containers and some other leftovers can create bunch of reclaimable storage. Docker prune commands do not have effect on resources (containers, volumes and networks) which are under use.\nYou can setup a cronjob which weekly prunes system, as provided below.\n$ crontab -e 0 0 * * 0 docker system prune -f # this will run at 00:00 on Sunday everyweek 0 0 * * 0 docker volume prune -f # will remove all (NOT in use) volumes weekly 0 0 * * 0 docker network prune -f # will remove all (NOT in use) networks You can write a script according to your needs then place it to cron job as well. Before adding cron jobs, make sure $USER has valid permissions for docker group.\nNow, we are sure that the error caused due to docker volumes which means we can proceed with a permanent solution.\nIf it the error is not caused due to docker volumes, it could be caused from running short of inodes or basically not enough storage area.\nPermanent Solution Permanent solution is using one of storage volume for storing docker volumes, instead of using root path. These are the steps that you may take;\nStop docker service !\n$ systemctl stop docker Rsync all data under /var/lib/docker to other directory under storage volume\n$ mkdir -p /data/mnt # creates plave to use for docker volumes $ rsync -a /var/lib/docker /data/mnt/ # will sync everything Rename default docker volumes path, this for taking a backup until we have success at the end.\n$ mv /var/lib/docker /var/lib/dockerbckp Create symbolic link to actual place\n$ ln -s /data/mnt/docker /var/lib/docker Enable DOCKER_OPTS\n$ vim /etc/default/docker DOCKER_OPTS=\u0026#34;--dns 8.8.8.8 --dns 8.8.4.4 -g /data/mnt/docker\u0026#34; Uncomment DOCEKR_OPTS add -g /data/mnt/docker as shown above in /etc/default/docker\nStart Docker daemon\n$ systemctl start docker If there was no error during implementation of steps, you can now test your setup.\nNote that thee should NOT be space between curly brackets when listing mount point only.\n$ docker volume create 2f2bd462b89c39bb641e7daf01048c5d811dd7796d7f89d250ea82c4532d2707 $ docker volume inspect --format { {.Mountpoint} } 2f2bd462b89c39b641e7daf01048c5d811dd7796d7f89d250ea82c4532d2707 /data/mnt/docker/volumes/2f2bd462b89c39bb641e7daf01048c5d811dd7796d7f89d250ea82c4532d2707/_data As you can observe above docker is using /data/mnt/docker for docker volumes, now we can safely remove old backup data. rm -rf /var/lib/dockerbckp\nI would like to mention that there is an information about changing docker volume instead implementing all steps mentioned above, just specifiying path under /etc/docker/daemon.js file would work (-example daemon.js is given-). However, when I tried that approach, it did not worked.\nExample\n$ vim /etc/docker/daemon.js { \u0026#34;data-root\u0026#34;: \u0026#34;/mnt/docker-data\u0026#34;, \u0026#34;storage-driver\u0026#34;: \u0026#34;overlay2\u0026#34; } Check this documentation: https://docs.docker.com/config/daemon/systemd/\nI hope, this post can help others to find required information quickly and implement necessary steps to get to work.\nReferences: https://github.com/moby/moby/issues/10613 https://github.com/moby/moby/issues/10613 https://success.docker.com/article/error-message-no-space-left-on-device-in-default-machine https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169 ","permalink":"https://mrturkmen.com/posts/no-space-left-on-device/","summary":"\u003ch1 id=\"summary\"\u003eSummary\u003c/h1\u003e\n\u003cp\u003eIn this post, one of the well known (-open to discuss :)-) error of \u0026ldquo;No space left on device\u0026rdquo; \u003cstrong\u003ewhich is caused due to Docker\u003c/strong\u003e will be solved with different approaches.\u003c/p\u003e\n\u003cp\u003eNote: \u0026ldquo;No space left on device\u0026rdquo; error can be caused due to any other reason than docker itself. Hence, it would be nice to make sure that the error is caused due to docker volumes.\u003c/p\u003e\n\u003cp\u003eYou can check whether it is caused due to docker volumes or not by following steps over \u003ca href=\"#ensure-about-the-problem\"\u003ehere\u003c/a\u003e\u003c/p\u003e","title":"error: no space left on this device "},{"content":"Giriş Elasticsearch üzerinde büyük boyuttaki verileri hızlı bir şekilde işlemek çaba gerektiren işlerden biridir. Bu yazıda bir çalışma esnasında yapılan elasticsearch performans iyileştirmelerini ve nasıl yapıldığını anlatmaya çalışacağım.\nBu iyileştirme işlemlerinin nasıl yapıldığına geçmeden once elasticsearch mimarisinde bulunan bazı bileşenlerden bahsetmekte yarar var.\nCluster : Elasticsearch bir veya birden fazla bilgisayarda entegre şekilde çalışabilir ve bu elasticsearch ün çalıştığı makinelere NODE denir. Cluster (Küme) ise bu node’ların oluşturduğu gruba verilen yapıya denir. Index : Elasticsearch üzerinde veriler indexlerde tutulur, index basit olarak dökümanların toplandığı ve tutulduğu yapıdır. Shard: Elasticsearch ü birden fazla makine üzerinde (sanal veya fiziksel makine) tutabildiğimizden dolayı, indekslerde tutulan veriler bu cluster adı verdiğimiz ortamlarda dağıtık (distributed) şekilde tutulur. Bu işlemin yönetim kısmını elasticsearch otomatik olarak halleder. Replica: Elasticsearch normalde (default) her indeks için 5 ana shard ve 1 replica oluşturur, yani her bir indeks 5 adet shard’a sahip ve her shard bir replica içermektedir. Aşağıda bu durumu gösteren bir ekran görüntüsü verilmiştir. http://\u0026lt;elk-ip\u0026gt;:9200/_cat/shards Bu performans iyileştirme adımları, tek sunucu (node) üzerinde çalışan Elasticsearch üzerinde yapılmıştır, yani dağıtık bir sistem üzerinde iyileştirme yapmak buradaki anlatılacaklardan farklı olacaktır. (Bazı kısımları benzerlik gösterse dahi)\nNot : Bu iyileştirme işlemleri, çalıştırılan sunucu sayısına (dağıtık sistem ise), internet hızına (dağıtık sistem ise), sunucuda çalışan işletim sisteminden, kullandığı disk, CPU ve RAM kapasitesine göre değişiklik gösterebilir.\nParalel Bulk indeksleme Elasticsearch üzerinde indeksleme işlemi birkaç farklı yöntem ile yapılabilmektedir bunlardan bazıları, tek tek indeksleme, bulk indeksleme ve parallel indekslemedir.\nTek tek indeksleme yöntemi, tahmin edeceğiniz üzere veri büyük olduğunda tercih edilecek bir yöntem değildir, nedeni ise her kayıt için elasticsearche istekte bulunmasından dolayıdır. Yani 10000 adet satır için 10000 istek gönderilecek demektir, bunun yerine bulk indeksleme tercih edilir 10000 adet kayıt için tek istek gönderimi yapar böylece hem istek sayısı minimuma indirilmiş olur, hem de indeksleme süresi azaltılmış olur. Bunun bir adım daha gelişmişi ise paralel bulk tır, bu indeksleme yönteminde ise birden fazla thread ile veri elasticsearche gönderilecektir sağlar. Bizim çalışmamızda paralel bulk işlemi kullanılmıştır.\nBu çalışmada, Elasticsearch ün Python modülleri kullanılmıştır, bu modülde paralel bulk kullanımı aşağıdaki şekildedir.\nParalel bulk kullanabilmek için Python generator kullanmak tercih edilen yöntemlerden biridir, nedeni hem ram kullanımı az olur, hemde tekrarlı (iterate) bir yapıya sahiptir.\nÖrnek generator yapısı :\ndef gendata(docs_list): for json in docs_list: yield { \u0026#34;_index\u0026#34;: \u0026#34;herhangibirsey\u0026#34;, \u0026#34;_type\u0026#34;: \u0026#34;_doc\u0026#34;, \u0026#34;isim\u0026#34;:json[\u0026#39;isim\u0026#39;], \u0026#34;soyisim\u0026#34;:json[\u0026#39;soyisim\u0026#39;], \u0026#34;sehir\u0026#34;: json[\u0026#39;sehir\u0026#39;], \u0026#34;yas\u0026#34;:json[\u0026#39;yas\u0026#39;], \u0026#34;meslek\u0026#34;:json[\u0026#39;meslek\u0026#39;] } Bu generator yapısında, gendata fonksiyonu docs_list adında bir liste alıyor ve bu listenin içeriği şu şekilde olduğunu varsayıyoruz:\ndocs_list= [{\u0026#34;isim\u0026#34;: \u0026#34;Mehmet\u0026#34;,\u0026#34;soyisim\u0026#34;: \u0026#34;Ataklar\u0026#34;,\u0026#34;sehir\u0026#34;: \u0026#34;Kocaeli\u0026#34;,\u0026#34;yas\u0026#34;: 45,\u0026#34;meslek\u0026#34;: \u0026#34;Ogretmen\u0026#34;}] gendata fonksiyonu docs_list listesi icerisindeki her bir dokumandan gereken alanlari alarak indeksleme fonksiyonuna vermektedir. Parallel bulk, Python script üzerinden şu şekilde çağrılabilir.\nfor response in parallel_bulk(elasticDeamon, gendata(doc_records), thread_count=7): pass Indeks yenileme aralığını kaldırma (refresh_interval) Node üzerinde bulunan indeks e, bulk indexleme işlemi yapılırken, indeks yenileme aralığı bulk indeksleme süresi boyunca ortadan kaldırılmalıdır. Çünkü elasticsearch ün her yenileme yapması sunucu üzerinde segment oluşturmasını sağlamaktadır, bu hem makinen kaynaklarına dezavantaj olarak yansımaktadır, ram ve cpu kullanımını artıran pahalı bir işlemdir.\nKibana üzerinde bulunan “Dev Tools” kısmından aşağıdaki verilen komut ile kaldırılabilir.\nPUT /\u0026lt;indeks-ismi\u0026gt;/_settings { \u0026#34;index\u0026#34;: { \u0026#34;refresh_interval\u0026#34;: -1 } } Terminal üzerinden:\ncurl -X PUT \u0026#34;\u0026lt;elk-ip\u0026gt;:9200/\u0026lt;index-ismi\u0026gt;/_settings\u0026#34; -H \u0026#39;Content-Type: application/json\u0026#39; -d\u0026#39; { \u0026#34;index\u0026#34; : { \u0026#34;refresh_interval\u0026#34; : -1 } } \u0026#39; Bulk indeksleme işlemi sona erdiğinde ise, aynı komutlar kullanılarak, yenileme aralığı “null” a eşitlenebilir. Böylece kullanıcı kibana üzerinden, yenileme aralığını kendisi ayarlayabilir.\nPUT /\u0026lt;index-ismi\u0026gt;/_settings { \u0026#34;index\u0026#34;: { \u0026#34;refresh_interval\u0026#34;: null } } Indeks kopyalarını devre dışı bırakmak (Replica) Kulağa hoş gelmesede indeks kopyalarını (replicas) devre dışı bırakmak indeksleme hızını artırır, en büyük dezavantajı indeksi herhangi bir hata durumunda veri kaybına karşı savunmasız bırakır.\nKibana “DevTools” kısmından kopyaların devre dışı bırakılması.\nPUT /\u0026lt;indeks-ismi\u0026gt;/_settings { \u0026#34;index\u0026#34; : { \u0026#34;number_of_replicas\u0026#34; : 0 } } Terminal üzerinden:\ncurl -X PUT \u0026#34;\u0026lt;elk-ip\u0026gt;:9200/\u0026lt;index-ismi\u0026gt;/_settings\u0026#34; -H \u0026#39;Content-Type: application/json\u0026#39; -d\u0026#39; { \u0026#34;index\u0026#34; : { \u0026#34;number_of_replicas\u0026#34; : 0 } } \u0026#39; Swap alanını kaldırmak. (Sunucu üzerindeki) Elasticsearchü hızlı yapan faktörlerden en önemlisi ram üzerinden işlem yapmasıdır. Linux sunucularında bulunan swap alanı, ram de yeterli alan kalmadığında veya ram üzerinde uzun süre işlem yapılmayan (aktif olmayan) dosyaların disk üzerinde kısa süreliğine saklanmasından oluşan alandır. Bu elasticsearh için dezavantaj olabilmektedir, elasticsearchün tamamen ram üzerinden işlem yapmasını sağlamak adına swap alanını kaldırmak indeksleme ve arama yapma hızını artıracaktır.\nSwap alanını geçici olarak şu şekilde kaldırabilirsiniz, terminal üzerinden bu komutu yazmanız yeterlidir.\n$ swapoff -a Swap alanını tamamen kaldırabilmek için “root” yetkisi ile /etc/fstab dosyası içerisinde swap kelimesi geçen kısmı yorum satırı yapmanız yeterli olacaktır.\nSwap alanını ortadan kaldırdıktan sonra sunucu üzerinde çalışan elasticsearch ayarlarında ufak bir değişiklik yapmak gerekecektir.\n/etc/elasticsearch/elasticsearch.yml\nelasticsearch.yml dosyası içerisine şu parametreyi eklemeniz gerekmektedir.\nbootstrap.mlockall: true Bu işlem elasticsearch ün tamamen RAM üzerinden işlem yapmasını sağlayacaktır.\nJVM Heap Alanını Artırmak Elasticsearch JVM heap, verileri hızlı bir şekilde işlemek ve veriler üzerindeki işlemleri yapabilmek için elasticsearche özel olarak ayrılmış bir alan. Bu alan normalde (default olarak) 1 GB alana sahiptir, eğer sunucu üzerinde yeterli miktarda RAM mevcut ise bu alanı artırmak indeksleme ve işlem yapma hızını artıracaktır.\nBurada önemli olan JVM Heap alanı 64 Bit yapıya sahip bir sunucu için maksimum 32 GB a kadar artırılmalıdır, sunucu üzerinde çok daha fazla RAM olsa dahi 32 GB limiti geçmemek gerekmektedir. Bununla ilgili detaylı açıklamaya buradan erişebilirsiniz: https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html\nJVM Heap alanı ayarlanırken genellikle fiziksel ram boyutunun yarısı kadar heap alanı vermek tercih edilir, 32 GB boyutunu geçmeyecek şekilde.\nJVM Heap ayarları şu şekilde yapılabilir, Debian tabanlı bir işletim sisteminde elasticsearch ün bulunduğu dizin altında jvm.options adında bir dosya bulunmaktadır.\nEğer heap alanını 16 GB ayarlamak isterseniz(fiziksel RAM in en az 32 GB olduğundan emin olunuz ), jvm.options dosyası içerisine şu şekilde kaydedebilirsiniz.\n/etc/elasticsearch/jvm.options ## bu jvm.options dosyası içerisine aşağıda verilen parametler girilir. -Xms16GB -Xmx16GB Bu parametreler, jvm.options dosyası içerisine kaydedildikten sonraki adımda ise elasticsearch servisini yeniden başlatmayı unutmayınız.\nsudo service elasticsearch restart SSD veya RAID 0 disk kullanımı HDD disklere göre çok hızlı olan SSD diskler, elasticsearch ün veriyi daha hızlı işlemesine, verimliliği artırmasına direkt olarak etki edecektir. RAID diskleri kullanırken RAID 0 haricindeki tiplerini kullanmak tercih edilmez.\nBu kısımda elasticsearch performansını artırmak için yapılması gerekli olabilecek bazı adımlardan bahsedildi bunlar özet olarak.\nParalel bulk indekslemek JVM heap alanı artırmak İndeks kopyaları devre dışı bırakmak İndeks yenileme aralığını devre dışı bırakmak Sunucu Swap alanını kaldırmak SSD veya RAID 0 Disk Kullanmak Bu, elasticsearch performans iyileştirme adımlarını gösteren birinci kısım, ikinci kısımda, elasticsearch üzerinde indeks oluştururken, mapping sisteminin verimize göre nasıl yapılandırılması gerektiğinden, indeks üzerinde otomatik olarak oluşturulan bazı alanların kaldırılmasından, optimum shard sayısının belirlenmesinden, indeks performans (benchmarking) ölçümlerinden ve Grafana üzerinden elasticsearch değerlerinin (CPU,I/O, RAM, DISK kullanımının) izlenmesi anlatılacaktır.\nBu çalışma esnasında yararlanılan kaynaklar https://blog.codecentric.de/en/2014/05/elasticsearch-indexing-performance-cheatsheet\u0026gt;\nhttps://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html\nhttps://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.htm\nhttps://elasticsearch-py.readthedocs.io/en/master\n","permalink":"https://mrturkmen.com/posts/elasticsearch-performans-art%C4%B1r%C4%B1m%C4%B1/","summary":"\u003ch4 id=\"giriş\"\u003eGiriş\u003c/h4\u003e\n\u003cp\u003eElasticsearch üzerinde büyük boyuttaki verileri hızlı bir şekilde işlemek çaba gerektiren işlerden biridir. Bu yazıda bir çalışma esnasında yapılan elasticsearch performans iyileştirmelerini ve nasıl yapıldığını anlatmaya çalışacağım.\u003c/p\u003e\n\u003cp\u003eBu iyileştirme işlemlerinin nasıl yapıldığına geçmeden once elasticsearch mimarisinde bulunan bazı bileşenlerden bahsetmekte yarar var.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eCluster\u003c/strong\u003e : Elasticsearch bir veya birden fazla bilgisayarda entegre şekilde çalışabilir ve bu elasticsearch ün çalıştığı makinelere NODE denir. Cluster (Küme) ise bu node’ların oluşturduğu gruba verilen yapıya denir.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eIndex\u003c/strong\u003e : Elasticsearch üzerinde veriler indexlerde tutulur, index basit olarak dökümanların toplandığı ve tutulduğu yapıdır.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eShard\u003c/strong\u003e: Elasticsearch ü birden fazla makine üzerinde (sanal veya fiziksel makine) tutabildiğimizden dolayı, indekslerde tutulan veriler bu cluster adı verdiğimiz ortamlarda dağıtık (distributed) şekilde tutulur. Bu işlemin yönetim kısmını elasticsearch otomatik olarak halleder.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eReplica\u003c/strong\u003e: Elasticsearch normalde (default) her indeks için 5 ana shard ve 1 replica oluşturur, yani her bir indeks 5 adet shard’a sahip ve her shard bir replica içermektedir.  Aşağıda bu durumu gösteren bir ekran görüntüsü verilmiştir.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ehttp://\u0026lt;elk-ip\u0026gt;:9200/_cat/shards\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cimg loading=\"lazy\" src=\"../../images/figure_1.png\"\u003e\u003c/p\u003e","title":"elasticsearch: performans artırımı "},{"content":"Özet Terminal üzerinden kullanılan en meşhur yazı düzenleme programı VIM hakkında komutlar ve bazı kullanışlı linux komutları\nScript nasıl yazılır. Dosya oluşturulur ve dosyanın başına terminalin yolu eklenir #!/bin/bash\nterminal komutlarını bu yolun altına yazınız.\ndaha sonra dosyanın iznini ayarlayınız. chmod +x dosya_ismi\nterminal scriptini şu şekilde çalıştırılabilirsiniz. ./dosya_ismi\nörnek bash scripti\n!/bin/bash echo \u0026#34;Adınzı giriniz: \u0026#34; read isim echo \u0026#34;Sifrenizi giriniz\u0026#34; read sifre if [[ ( $isim == \u0026#34;admin\u0026#34; \u0026amp;\u0026amp; $sifre == \u0026#34;random\u0026#34; ) ]]; then echo \u0026#34;Başarılı\u0026#34; else echo \u0026#34;Başarısız\u0026#34; fi VIM Birden fazla dosya üzerinde çalışmak $ vim *.txt # birden fazla txt ile dosyasını aynı anda açmanızı sağlar (eğer birden fazla dosya mevcut ise) $ :wall veya :qall # bütün açık dosyalardan yaz veya çık komutudur. $ vim -o *.txt # birden fazle txt dosyasını açar ve yatay düzlemde gösterir, dikey düzlem için -O parametresi kullanılara $ :args *.txt # txt ile biten bütün dosyaları argument listesine aktarır. $ :all # bütün dosyaları yatay düzlemde ayırır $ CTRL-w # birden fazla pencere arasında gezmenizi sağlar $ :split # aynı dosyayı iki farklı pencerede gösterir. $ :split \u0026lt;acılacak_dosya\u0026gt; # dosyayı yeni bir pencerede açar $ :vsplit # brden fazla pencereyi dikey komunda ayırır, tablolar için çok kullanışlıdır. \u0026#34;:set scrollbind \u0026#34; komutu ile açık dosyalar da aynı anda yukarı aşağı yapabilirsiniz. $ :close # bulunduğunuz pencereyi kapatır $ :only # bulunduğunuz pencere hariç diğerlerinin tamamını kapatır. Hece Kontrolü \u0026amp; Sözlük $ aspell -c \u0026lt;dosya\u0026gt; # verilen dosyada heceleri kontrol eder, terminal komutudur $ aspell -l \u0026lt;dosya\u0026gt; # terminal komutu $ :! dict \u0026lt;cumle\u0026gt; # cümlenin anlamını kontrol etmenizi sağlar $ :! wn \u0026#39;cumle\u0026#39; -over # cümlenin eş anlamlılarını gösterir Dosyayı yazdırma $ :ha # bütün dosyayı yazdırır $ :#,#ha # (#,#) ile belirtilen alandaki metini yazdırır Birleştime / Ekleme Komutu $ :r \u0026lt;dosya_ismi\u0026gt; # açık olan dosya içerisine, # aynı dizinde olan başka bir dosyayı eklemek için bu komut kullanılabilir, # imlecin hizasından sonra ekleme yapar Geri Alma / Yeniden Alma $ u # en son yaptığınız değişikliği geri alır $ U # yaptığınız bütün değişiklikleri geri alır $ CTRL-R # geri alınmış bir kısmı yeniden getirmenizi sağlar. Kopyalama \u0026amp; Yapıştırma $ yy # imlecin bulunduğu satırı kopyalar, 2 satır kopyalamak için 2yy kullanılabilir. $ p # kesilen/kopyalanan içeriği imleçten başlayacak şekilde yapıştırır Silme/Kesme (NORMAL modda uygulanır. Yani Vim komut satırında değil. EXE modunda değil. ) $ x # imlecin üzerinde bulunduğu karakteri siler. $ dw # imlecin bulunduğu kelimeyi sonuna kadar siler (Boşluklar dahil ) $ de # imlecin bulunduğu kelimeyi sonuna kadar siler (Boşluklar hariç ) $ cw # kelimenin geriye kalan kısmını siler ve sizi ekleme moduna alır, ekleme modundan ESC ile çıkabilirsiniz. $ c$ # bulunduğu satırı tamamen siler ve sizi ekleme moduna alır ekleme modundan ESC ile çıkabilirsiniz. $ d$ # imlecten itibaren satırı siler e $ dd # satırı tamamen siler, imlecin nerede olduğunun önemi yoktur $ 2dd # ileriki 2 satırı siler, benzer sekilde 3dd : uc satır siler, 4dd: dort satır siler, (imlecten bagımsız) Koyma $ p # kesilen/kopyalanan içeriği imleçten başlayacak şekilde yapıştırır Dosya içerisinde arama (Vim) (bu kısımda genelde düzenli ifadeler kullanılır ) $ /aramak_istediğiniz_düzen # yazdığınız ifadeyi açık olan belge içerisinde arar ve hepsini işaretler $ ?aramak_istediğiniz_düzen # yazdığınız ifadeyi açık olan belge içerisinde arar ama işaretlemez, n ile ileriki kelimeyi görebilirsiniz. $ :set ic # kelimelerin büyük/küçük harf ayrımını ortadan kaldırır $ :set hls # aranan ve bulunan kelimeleri vurgulu şekilde gösterir. Düzenli ifadeler ile metin yönetimi $ :s/harf1/harf2/ # harf1, harf2 ile değiştirilir fakat sadece ilk karşılaşmada yapılır $ :s/harf1/harf2/g # bütün dosya içerisindeki harf1, harf2 ile değiştirilir. $ :s/harf1/harf2/gc # yukarıdaki işlemin aynısını onay alarak yapmak için \u0026#34;c\u0026#34; eklenir $ :#,#s/harf1/harf2/g # (#,#) arasındaki satırlarda bulunan harf1, harf2 ile değiştirilir. $ :%s/harf1/harf2/g # tüm dosyadaki harf1 ifadesi harf2 ile değiştirilir. $ :%s/\\(harf1\\)\\(.*\\)/\\1/g # harf1 sonrakisindeki bütün satırları siler. $ :%s/\\(SL\\dm\\d\\d\\d\\d\\d\\.\\d\\)\\(.*\\)/\\1\\t\\2/g # SL1m12345.1 ve tanımı arasına TAB boşluğu ekler $ :%s/\\n/ifade/g #Satır verilen ifade ile değiştirilir. $ :%s/\\(^SL\\dm\\d\\d\\d\\d\\d.\\d\\t.\\{-}\\t.\\{-}\\t.\\{-}\\t.\\{-}\\t\\).\\{-}\\t/\\1/g # 5 ve 6.ncı TAB taki (5. Kolondaki), içeriği \u0026#34;{-}\u0026#34; ile degiştirir. $ :#,#s/\\( \\{-} \\|\\.\\|\\n\\)/\\1/g # (#,#) verilen aralıkta ne kadar cümle olduğunu hesaplar $ :%s/\\(E\\{6,\\}\\)/\u0026lt;font color=\u0026#34;green\u0026#34;\u0026gt;\\1\u0026lt;\\/font\u0026gt;/g # 6 dan fazla E geçen kısımları, HTML renkleri ile vurgular. $ :%s/\\([A-Z]\\)/\\l\\1/g # Büyük harfleri, küçük harfler ile değiştirir, \u0026#39;%s/\\([A-Z]\\)/\\u\\1/g\u0026#39; , bu ise küçük harfleri büyük harfler ile değiştirir. $ :g/ifade/ s/\\([A-Z]\\)/\\l\\1/g | copy $ # ifade yeni oluşturulan ifade ile değiştirilir eşdeğer olanlar copy $ ile yazdırılır. HTML Düzenleme -metini HTML formatına cevirme $ :runtime! syntax/2html.vim # vim içerisinde bu komutu çalıştırınız. Vim içerisinden terminal komutu çalıştırma $ :!\u0026lt;terminal_komutu\u0026gt; \u0026lt;ENTER\u0026gt; # terminal komutunu vim içerisinden çalıştırır $ :sh terminal ile vim arasında gezmenizi sağlar Tablo düzenleyicisi olarak Vim\u0026rsquo; i kullanmak $ v # karakterleri seçmek için görsel mod başlatılır. $ V # satırları seçmek için görsel mod başlatılır. $ CTRL-V # blok görsel seçim yapmanızı sağlar. $ :set scrollbind # aynı ayna ayrılan iki ayrı dosyada gezinti yapmanızı sağlar. Vim ayarlarını değiştirmek - .vimrc dosyası içerisindeki parametreler isteğinize göre değiştirilebilir.\nKullanışlı terminal komutları $ cat \u0026lt;dosya1\u0026gt; \u0026lt;dosya2\u0026gt; \u0026gt; \u0026lt;sonuc\u0026gt; # dosya1 ve dosya2 yi sonuc dosyasina kopyalar ve sonuc dosyasini olusturur. $ paste \u0026lt;dosya1\u0026gt; \u0026lt;dosya2\u0026gt; \u0026gt; \u0026lt;p_sonuc\u0026gt; # iki farklı kaynaktan gelen girdiyi, aralarında TAB boşluğu olacak şekilde aynı dosya (p_sonuc) içerisine yapıştırır. $ cmp \u0026lt;dosya1\u0026gt; \u0026lt;dosya2\u0026gt; # iki dosyanın aynı olup olmadıgını size bildirir. $ diff \u0026lt;dosya1\u0026gt; \u0026lt;dosya2\u0026gt; # iki dosya arasındaki farklılıkları gösterir $ head -\u0026lt;numara\u0026gt; \u0026lt;dosya\u0026gt; # verdiğiniz numara kadar ilk X satırı yazdırır. $ tail -\u0026lt;numara\u0026gt; \u0026lt;dosya\u0026gt; # verdiğiniz numara kadar son X satırı yazdırır. $ split -l \u0026lt;numara\u0026gt; \u0026lt;dosya\u0026gt; # dosyanın satırılarını ayırır. $ csplit -f out dosya_ismi \u0026#34;%^\u0026gt;%\u0026#34; \u0026#34;/^\u0026gt;/\u0026#34; \u0026#34;{*}\u0026#34; # dosya_ismini \u0026gt; den itibaren birçok farklı küçük dosyalar oluşturur. $ sort \u0026lt;dosya_ismi\u0026gt; # dosya içerisindekileri sıralar -b argument kullanılırsa boşlukları yok sayar. $ sort -k 2,2 -k 3,3n girdi_dosyası \u0026gt; cıktı # -k argument i kolon için, -n sayısal olarak sıralar ve tablo şeklinde kaydeder. $ sort girdi_dosyası | uniq \u0026gt; cıktı # uniq komutu aynı olan verileri dahil etmez. $ join -1 1 -2 1 \u0026lt;tablo1\u0026gt; \u0026lt;tablo2\u0026gt; # tablo1 ve tablo2 yi birleştirir, -1 dosya1, 1:kolon1; -2dosya2, col2. $ sort tablo1 \u0026gt; tablo1a; sort tablo2 \u0026gt; tablo2a; join -a 1 -t \u0026#34;`echo -e \u0026#39;\\t\u0026#39;`\u0026#34; tablo1a tablo2a \u0026gt; tablo3 # \u0026#39;-a \u0026lt;tablo\u0026gt;\u0026#39; : verilen tablonun bütün kayıtlarını yazdırır. Normalde yazdırma işlemi iki tabloda ortak olan kısımları yazdırır. \u0026#39;-t \u0026#34;`echo -e \u0026#39;\\t\u0026#39;`\u0026#34; -\u0026gt;\u0026#39; : TAB boşluğu kullanarak tabloları çıktı dosyasına yazdırır. $ cat tablom | cut -d , -f1-3 # cut komutu : tablonun belirlenen kısımları alır, -d alanların nasıl ayrılacağını belirtilsiniz. -d : burada , olarak belirlenmiştir, normalde TAB boşluk, -f tablonun kolonlarını belirtir, kolon 1 den 3 e. Kullanışlı tek satır komutlar $ for i in *.input; do mv $i ${i/isim\\.eski/isim\\.yeni}; done # isim.eski adındaki dosyanın ismini, isim.yeni olarak değiştirir. Komutu test etmek için, do mv komutu önüne \u0026#34;echo\u0026#34; konulabilir. $ for i in *.girdi; do ./uygulama $i; done # bir çok dosya için verilen uygulamayı çalıştırır. $ for i in *.girdi; do komut -d /veri/../veri_tabanı -i $i \u0026gt; $i.out; done # komut for döngüsü içerisinde *.girdi üzerinde çalışır ve *.out dosyası oluşturur. $ for i girdi *.pep; do hedef -db /usr/../veri_tabanı -seed $i -out $i; done # hedef in üzerinde for döngüsü çalıştırılır ve çıktı dosyası yazdırılır. $ for j girdi 0 1 2 3 4 5 6 7 8 9; do grep -iH \u0026lt;ifade\u0026gt; *$j.seq; done # verilen ifadeyi girdi \u0026gt; 10.000 dosyaya kadar arar ve ne kadar o ifade geçtiğini yazdırır. $ for i in *.pep; do echo -e \u0026#34;$i\\n\\n17\\n33\\n\\n\\n\u0026#34; | ./program $i \u0026gt; $i.out; done # etkileşimli programı çalıştırır ve girdi/çıktı sorar. Basit Perl Komutları $ perl -p -i -w -e \u0026#39;s/ifade1/ifade2/g\u0026#39; girdi_dosyası # girdi dosyası içerisindekileri verilen ifadelere göre değişimini yapar. \u0026#39;-p\u0026#39; bu komut yedek bir dosya oluşturur $ perl -ne \u0026#39;print if (/ifade1/ ? ($c=1) : (--$c \u0026gt; 0)) ; print if (/ifade2/ ? ($d = 1) : (--$d \u0026gt; 0))\u0026#39; girdi_dosyası \u0026gt; cıktı_dosyası # ifade1 ve ifade2 içeren satırları ayrıştırır (parse eder.) WGET (terminal üzerinden linki verilen dosya indirimini gerçekleştirir.) $ wget ftp://ftp.itu.edu.tr.... # verilen linkteki dosya wget komutunun çalıştırıldığı dizine iner. SCP (İki makine arasında güvenli kopyalama işlemi sağlar. ) Genel Kullanım. $ scp kopyalanacak_dosya kopyalanacak_yer # Örnekler Sunucudan dosya kopyalamak için (bilgisayarınızın terminalinden) $ scp kullanıcı@sunucu_ip:dosya_adı . # \u0026#39;.\u0026#39; en sona nokta koyulması, sunucu üzerindeki kopyalanacak dosyayı bulunduğunuz yere kopyalamasını sağlar. Bilgisayarınızdan sunucuya kopyalama yapmak için. (Bilgisayar terminalinden) $ scp resim.jpg kullanıcı@sunucu_ip:~/belgeler/resimler/ Sunucu üzerinde bulunan klasörü bilgisayarımıza kopyalamak için. (Bilgisayar terminalinden) $ scp -r kullanıcı@sunucu_ip:dizin/ ~/Masaustu Bilgisayar üzerinde bulunan klasörü sunucuya kopyalamak için. (Bilgisayar terminalinden) $ scp -r klasör/ kullanıcı@sunucu_ip:dizin/ NFTP : (Dosya transfer işlemlerinizi kolay şekilde terminal üzerinden yapmanızı sağlar ) $ open ncftp $ ncftp\u0026gt; open sunucu_url # sunucuya baglantı sağlanıyor.. $ ncftp\u0026gt; cd /root/Masaustu. # masaustune gecildi $ ncftp\u0026gt; get resimler.gz # masaustunde bulunan resimler.gz indirildi. $ ncftp\u0026gt; bye # gule gule mesajı alındı ","permalink":"https://mrturkmen.com/posts/vim/","summary":"\u003ch2 id=\"özet\"\u003eÖzet\u003c/h2\u003e\n\u003cp\u003eTerminal üzerinden kullanılan en meşhur yazı düzenleme programı VIM hakkında komutlar ve bazı kullanışlı linux komutları\u003c/p\u003e\n\u003ch2 id=\"script-nasıl-yazılır\"\u003eScript nasıl yazılır.\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eDosya oluşturulur ve dosyanın başına terminalin yolu eklenir \u003ccode\u003e#!/bin/bash\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eterminal komutlarını bu yolun altına yazınız.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003edaha sonra dosyanın iznini ayarlayınız. chmod +x dosya_ismi\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eterminal scriptini şu şekilde çalıştırılabilirsiniz. ./dosya_ismi\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eörnek bash scripti\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e!/bin/bash\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eecho \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;Adınzı giriniz: \u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eread isim\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eecho \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;Sifrenizi giriniz\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eread sifre\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e[[\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e(\u003c/span\u003e $isim \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;admin\u0026#34;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e $sifre \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;random\u0026#34;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e]]\u003c/span\u003e; \u003cspan style=\"color:#66d9ef\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eecho \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;Başarılı\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003eelse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eecho \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;Başarısız\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"vim\"\u003eVIM\u003c/h2\u003e\n\u003ch4 id=\"birden-fazla-dosya-üzerinde-çalışmak\"\u003eBirden fazla dosya üzerinde çalışmak\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ vim *.txt \u003cspan style=\"color:#75715e\"\u003e# birden fazla txt ile dosyasını aynı anda açmanızı sağlar (eğer birden fazla dosya mevcut ise)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :wall veya :qall \u003cspan style=\"color:#75715e\"\u003e# bütün açık dosyalardan yaz veya çık komutudur.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ vim -o *.txt \u003cspan style=\"color:#75715e\"\u003e# birden fazle txt dosyasını açar ve yatay düzlemde gösterir, dikey düzlem için -O parametresi kullanılara\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :args *.txt \u003cspan style=\"color:#75715e\"\u003e# txt ile biten bütün dosyaları argument listesine aktarır.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :all \u003cspan style=\"color:#75715e\"\u003e# bütün dosyaları yatay düzlemde ayırır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ CTRL-w \u003cspan style=\"color:#75715e\"\u003e# birden fazla pencere arasında gezmenizi sağlar\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :split \u003cspan style=\"color:#75715e\"\u003e# aynı dosyayı iki farklı pencerede gösterir.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :split \u0026lt;acılacak_dosya\u0026gt; \u003cspan style=\"color:#75715e\"\u003e# dosyayı yeni bir pencerede açar\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :vsplit \u003cspan style=\"color:#75715e\"\u003e# brden fazla pencereyi dikey komunda ayırır, tablolar için çok kullanışlıdır. \u0026#34;:set scrollbind \u0026#34; komutu ile açık dosyalar da aynı anda yukarı aşağı yapabilirsiniz. \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :close \u003cspan style=\"color:#75715e\"\u003e# bulunduğunuz pencereyi kapatır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :only \u003cspan style=\"color:#75715e\"\u003e# bulunduğunuz pencere hariç diğerlerinin tamamını kapatır. \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"hece-kontrolü--sözlük\"\u003eHece Kontrolü \u0026amp; Sözlük\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ aspell -c \u0026lt;dosya\u0026gt; \u003cspan style=\"color:#75715e\"\u003e# verilen dosyada heceleri kontrol eder, terminal komutudur\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ aspell -l \u0026lt;dosya\u0026gt; \u003cspan style=\"color:#75715e\"\u003e# terminal komutu\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :! dict \u0026lt;cumle\u0026gt; \u003cspan style=\"color:#75715e\"\u003e# cümlenin anlamını kontrol etmenizi sağlar\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :! wn \u003cspan style=\"color:#e6db74\"\u003e\u0026#39;cumle\u0026#39;\u003c/span\u003e -over \u003cspan style=\"color:#75715e\"\u003e# cümlenin eş anlamlılarını gösterir\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"dosyayı-yazdırma\"\u003eDosyayı yazdırma\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e $ :ha \u003cspan style=\"color:#75715e\"\u003e# bütün dosyayı yazdırır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e $ :#,#ha \u003cspan style=\"color:#75715e\"\u003e# (#,#) ile belirtilen alandaki metini yazdırır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"birleştime--ekleme-komutu\"\u003eBirleştime / Ekleme Komutu\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e $ :r \u0026lt;dosya_ismi\u0026gt; \u003cspan style=\"color:#75715e\"\u003e# açık olan dosya içerisine,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                   \u003cspan style=\"color:#75715e\"\u003e# aynı dizinde olan başka bir dosyayı eklemek için bu komut kullanılabilir,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                   \u003cspan style=\"color:#75715e\"\u003e# imlecin hizasından sonra ekleme yapar\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"geri-alma--yeniden-alma\"\u003eGeri Alma / Yeniden Alma\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ u \u003cspan style=\"color:#75715e\"\u003e# en son yaptığınız değişikliği geri alır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ U \u003cspan style=\"color:#75715e\"\u003e# yaptığınız bütün değişiklikleri geri alır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ CTRL-R \u003cspan style=\"color:#75715e\"\u003e# geri alınmış bir kısmı yeniden getirmenizi sağlar.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"kopyalama--yapıştırma\"\u003eKopyalama \u0026amp; Yapıştırma\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ yy \u003cspan style=\"color:#75715e\"\u003e# imlecin bulunduğu satırı kopyalar, 2 satır kopyalamak için 2yy kullanılabilir.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ p \u003cspan style=\"color:#75715e\"\u003e# kesilen/kopyalanan içeriği imleçten başlayacak şekilde yapıştırır \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"silmekesme-normal-modda-uygulanır-yani-vim-komut-satırında-değil-exe-modunda-değil-\"\u003eSilme/Kesme (NORMAL modda uygulanır. Yani Vim komut satırında değil. EXE modunda değil. )\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ x \u003cspan style=\"color:#75715e\"\u003e# imlecin üzerinde bulunduğu karakteri siler.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ dw \u003cspan style=\"color:#75715e\"\u003e# imlecin bulunduğu kelimeyi sonuna kadar siler (Boşluklar dahil )\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ de \u003cspan style=\"color:#75715e\"\u003e# imlecin bulunduğu kelimeyi sonuna kadar siler (Boşluklar hariç )\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ cw \u003cspan style=\"color:#75715e\"\u003e# kelimenin geriye kalan kısmını siler ve sizi ekleme moduna alır, ekleme modundan ESC ile çıkabilirsiniz.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ c$ \u003cspan style=\"color:#75715e\"\u003e#  bulunduğu satırı tamamen siler ve sizi ekleme moduna alır ekleme modundan ESC ile çıkabilirsiniz. \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ d$ \u003cspan style=\"color:#75715e\"\u003e# imlecten itibaren satırı siler e\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ dd \u003cspan style=\"color:#75715e\"\u003e# satırı tamamen siler, imlecin nerede olduğunun önemi yoktur\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ 2dd \u003cspan style=\"color:#75715e\"\u003e# ileriki 2 satırı siler, benzer sekilde 3dd : uc satır siler, 4dd: dort satır siler, (imlecten bagımsız)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e   Koyma  \n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ p \u003cspan style=\"color:#75715e\"\u003e# kesilen/kopyalanan içeriği imleçten başlayacak şekilde yapıştırır \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"dosya-içerisinde-arama-vim-bu-kısımda-genelde-düzenli-ifadeler-kullanılır-\"\u003eDosya içerisinde arama (Vim) (bu kısımda genelde düzenli ifadeler kullanılır )\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ /aramak_istediğiniz_düzen \u003cspan style=\"color:#75715e\"\u003e# yazdığınız ifadeyi açık olan belge içerisinde arar ve hepsini işaretler\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ ?aramak_istediğiniz_düzen \u003cspan style=\"color:#75715e\"\u003e# yazdığınız ifadeyi açık olan belge içerisinde arar ama işaretlemez, n ile ileriki kelimeyi görebilirsiniz. \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :set ic \u003cspan style=\"color:#75715e\"\u003e# kelimelerin büyük/küçük harf ayrımını ortadan kaldırır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :set hls \u003cspan style=\"color:#75715e\"\u003e# aranan ve bulunan kelimeleri vurgulu şekilde gösterir.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"düzenli-ifadeler-ile-metin-yönetimi\"\u003eDüzenli ifadeler ile metin yönetimi\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :s/harf1/harf2/ \u003cspan style=\"color:#75715e\"\u003e# harf1, harf2 ile değiştirilir fakat sadece ilk karşılaşmada yapılır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :s/harf1/harf2/g \u003cspan style=\"color:#75715e\"\u003e# bütün dosya içerisindeki harf1, harf2 ile değiştirilir.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :s/harf1/harf2/gc \u003cspan style=\"color:#75715e\"\u003e# yukarıdaki işlemin aynısını onay alarak yapmak için \u0026#34;c\u0026#34; eklenir\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :#,#s/harf1/harf2/g \u003cspan style=\"color:#75715e\"\u003e#  (#,#) arasındaki satırlarda bulunan harf1, harf2 ile değiştirilir.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :%s/harf1/harf2/g \u003cspan style=\"color:#75715e\"\u003e# tüm dosyadaki harf1 ifadesi harf2 ile değiştirilir.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :%s/\u003cspan style=\"color:#ae81ff\"\u003e\\(\u003c/span\u003eharf1\u003cspan style=\"color:#ae81ff\"\u003e\\)\\(\u003c/span\u003e.*\u003cspan style=\"color:#ae81ff\"\u003e\\)\u003c/span\u003e/\u003cspan style=\"color:#ae81ff\"\u003e\\1\u003c/span\u003e/g \u003cspan style=\"color:#75715e\"\u003e# harf1 sonrakisindeki bütün satırları siler.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :%s/\u003cspan style=\"color:#ae81ff\"\u003e\\(\u003c/span\u003eSL\u003cspan style=\"color:#ae81ff\"\u003e\\d\u003c/span\u003em\u003cspan style=\"color:#ae81ff\"\u003e\\d\\d\\d\\d\\d\\.\\d\\)\\(\u003c/span\u003e.*\u003cspan style=\"color:#ae81ff\"\u003e\\)\u003c/span\u003e/\u003cspan style=\"color:#ae81ff\"\u003e\\1\\t\\2\u003c/span\u003e/g \u003cspan style=\"color:#75715e\"\u003e#  SL1m12345.1 ve tanımı arasına TAB boşluğu ekler \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :%s/\u003cspan style=\"color:#ae81ff\"\u003e\\n\u003c/span\u003e/ifade/g \u003cspan style=\"color:#75715e\"\u003e#Satır verilen ifade ile değiştirilir.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :%s/\u003cspan style=\"color:#ae81ff\"\u003e\\(\u003c/span\u003e^SL\u003cspan style=\"color:#ae81ff\"\u003e\\d\u003c/span\u003em\u003cspan style=\"color:#ae81ff\"\u003e\\d\\d\\d\\d\\d\u003c/span\u003e.\u003cspan style=\"color:#ae81ff\"\u003e\\d\\t\u003c/span\u003e.\u003cspan style=\"color:#ae81ff\"\u003e\\{\u003c/span\u003e-\u003cspan style=\"color:#f92672\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e\\t\u003c/span\u003e.\u003cspan style=\"color:#ae81ff\"\u003e\\{\u003c/span\u003e-\u003cspan style=\"color:#f92672\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e\\t\u003c/span\u003e.\u003cspan style=\"color:#ae81ff\"\u003e\\{\u003c/span\u003e-\u003cspan style=\"color:#f92672\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e\\t\u003c/span\u003e.\u003cspan style=\"color:#ae81ff\"\u003e\\{\u003c/span\u003e-\u003cspan style=\"color:#f92672\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e\\t\\)\u003c/span\u003e.\u003cspan style=\"color:#ae81ff\"\u003e\\{\u003c/span\u003e-\u003cspan style=\"color:#f92672\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e\\t\u003c/span\u003e/\u003cspan style=\"color:#ae81ff\"\u003e\\1\u003c/span\u003e/g  \u003cspan style=\"color:#75715e\"\u003e#  5 ve 6.ncı TAB taki (5. Kolondaki), içeriği \u0026#34;{-}\u0026#34; ile degiştirir. \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :#,#s/\u003cspan style=\"color:#ae81ff\"\u003e\\(\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e\\{\u003c/span\u003e-\u003cspan style=\"color:#f92672\"\u003e}\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e\\|\\.\\|\\n\\)\u003c/span\u003e/\u003cspan style=\"color:#ae81ff\"\u003e\\1\u003c/span\u003e/g \u003cspan style=\"color:#75715e\"\u003e# (#,#) verilen aralıkta ne kadar cümle olduğunu hesaplar\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :%s/\u003cspan style=\"color:#ae81ff\"\u003e\\(\u003c/span\u003eE\u003cspan style=\"color:#ae81ff\"\u003e\\{\u003c/span\u003e6,\u003cspan style=\"color:#ae81ff\"\u003e\\}\\)\u003c/span\u003e/\u0026lt;font color\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#e6db74\"\u003e\u0026#34;green\u0026#34;\u003c/span\u003e\u0026gt;\u003cspan style=\"color:#ae81ff\"\u003e\\1\u003c/span\u003e\u0026lt;\u003cspan style=\"color:#ae81ff\"\u003e\\/\u003c/span\u003efont\u0026gt;/g \u003cspan style=\"color:#75715e\"\u003e# 6 dan fazla E geçen kısımları, HTML renkleri ile vurgular.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :%s/\u003cspan style=\"color:#ae81ff\"\u003e\\(\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e[\u003c/span\u003eA-Z\u003cspan style=\"color:#f92672\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e\\)\u003c/span\u003e/\u003cspan style=\"color:#ae81ff\"\u003e\\l\\1\u003c/span\u003e/g \u003cspan style=\"color:#75715e\"\u003e# Büyük harfleri, küçük harfler ile değiştirir, \u0026#39;%s/\\([A-Z]\\)/\\u\\1/g\u0026#39; , bu ise küçük harfleri büyük harfler ile değiştirir.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :g/ifade/ s/\u003cspan style=\"color:#ae81ff\"\u003e\\(\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e[\u003c/span\u003eA-Z\u003cspan style=\"color:#f92672\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e\\)\u003c/span\u003e/\u003cspan style=\"color:#ae81ff\"\u003e\\l\\1\u003c/span\u003e/g | copy $ \u003cspan style=\"color:#75715e\"\u003e# ifade yeni oluşturulan ifade ile değiştirilir eşdeğer olanlar copy $ ile yazdırılır. \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"html-düzenleme\"\u003eHTML Düzenleme\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e-metini HTML formatına cevirme\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :runtime! syntax/2html.vim \u003cspan style=\"color:#75715e\"\u003e# vim içerisinde bu komutu çalıştırınız.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"vim-içerisinden-terminal-komutu-çalıştırma\"\u003eVim içerisinden terminal komutu çalıştırma\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :!\u0026lt;terminal_komutu\u0026gt; \u0026lt;ENTER\u0026gt; \u003cspan style=\"color:#75715e\"\u003e# terminal komutunu vim içerisinden çalıştırır\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :sh terminal ile vim arasında gezmenizi sağlar\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"tablo-düzenleyicisi-olarak-vim-i-kullanmak\"\u003eTablo düzenleyicisi olarak Vim\u0026rsquo; i kullanmak\u003c/h4\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ v \u003cspan style=\"color:#75715e\"\u003e# karakterleri seçmek için görsel mod başlatılır.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ V \u003cspan style=\"color:#75715e\"\u003e# satırları seçmek için görsel mod başlatılır.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ CTRL-V \u003cspan style=\"color:#75715e\"\u003e# blok görsel seçim yapmanızı sağlar.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e$ :set scrollbind \u003cspan style=\"color:#75715e\"\u003e# aynı ayna ayrılan iki ayrı dosyada gezinti yapmanızı sağlar. \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"vim-ayarlarını-değiştirmek\"\u003eVim ayarlarını değiştirmek\u003c/h4\u003e\n\u003cp\u003e\u003cstrong\u003e- .vimrc dosyası içerisindeki parametreler isteğinize göre değiştirilebilir.\u003c/strong\u003e\u003c/p\u003e","title":"vim "},{"content":"Özet: Bu yazıda linux ortamına biraz daha giriş yaparak, linux ortamında bulunan komutlar hakkında kısa bilgilendirme yapılması planlanmaktadır.\nGiriş Neden Linux ? Birden fazla işlemi aynı anda kolay şekilde yapmanızı sağlar Uzaktan işlemlerinizi halletmede büyük kolaylık sağlar Birden fazla kullanıcı aynı sunucuya erişebilir Terminale, bir sistem üzerinde olan kaynaklara birden fazla erişim mümkündür Arayüz olan sistemlere göre daha performanslı, Bedava , Güncel Temeller Bu bilgilendirme dosyası için not Bütün komutlar büyük ve küçük harfe duyarlıdır.\n\u0026quot;$\u0026quot;komutun başlangıcını temsil eder.\n\u0026quot;#\u0026quot; komutun sonunu temsil eder.\nWindows bilgisayar üzerinden giriş için PuTTY : windows bilgisayar üzerinden SSH ile baglantı sağlamak için gereklidir. PuTTY programında SSH ile bağlantı için sunucu IP adresi ve PORT numarasını bilmelisiniz. Mac veya Linux Bilgisayarlardan Erişim Bu tür bilgisayarlar UNIX tabanlı olduğundan dolayı terminal üzerinden aşağıda verilen komutları yazmanız yeterli olacaktır ekstradan herhangi bir programa gerek duyulmamaktadır.\n$ ssh \u0026lt;kullanıcı_adı\u0026gt;@\u0026lt;sunucu_adresi(IP)\u0026gt; $ kullanıcı_adı: ... $ şifre: ... {% endhighlight %} #### Giriş yaptıktan sonra şifrenizi değiştirmek isterseniz ```bash $ passwd # bu komutu kullanabilirsiniz. # bu komut sayesinde giriş yapılan # kullanıcı için yeni şifre # belirleyebilirsiniz. {% endhighlight %} #### Listeleme ```bash $ pwd # şu anda bulunduğunuz konumu çıktı olarak yazdırır $ ls # bulunduğunuz konumdaki dosyaları ve klasörleri listeler $ ll # ls komutunun eşdeşi olarak tanımlı bir ifadedir genelde \u0026#34;ls -alF\u0026#34; olarak kayıtlıdır bash profilinde $ ll -R # dosyalar/klasörler listelenir, klasörler içerisindeki dosyalarda listelenir $ ll -t # listeleme işlemi kronolojik şekilde gerçekleşir. $ stat \u0026lt;dosya_adı\u0026gt; # dosyaya ait bilgileri meta bilgileri listeler $ whoami # sizin sistem tarafından kim olduğunuzu söyler, yani kullanıcı adınızı listeler $ hostname # bağlı olduğunu makinenin URL ni yada IP sini gösterir Dosyalar ve Klasörler $ mkdir \u0026lt;klasör_adı\u0026gt; # belirlenen isimde klasör oluşturur $ cd \u0026lt;klasör_adı\u0026gt; # klasör adı tanımlanan klasöre gidersini. $ cd .. # üst klasöre gitmenizi sağlar $ cd ../../ # iki üst klasöre gitmenizi sağlar $ cd # ana klasörüne gidersiniz $ rmdir \u0026lt;klasör_adı\u0026gt; # klasörü siler $ rm \u0026lt;dosya_adı\u0026gt; # dosyayı siler $ rm -r \u0026lt;klasör_adı\u0026gt; # klasörü ve içerisindeki bütün dosyaları siler $ mv \u0026lt;dosyaadı1\u0026gt; \u0026lt;dosyaadı2\u0026gt; # isim değiştirmenizi sağlar, name $ mv \u0026lt;dosyaadı\u0026gt; \u0026lt;taşınacak_yol\u0026gt; # dosyayı belirtilen yere taşır $ cp \u0026lt;dosyaadı\u0026gt; \u0026lt;kopyalanacak_yol\u0026gt; # dosyayı belirtilen yere kopyalar, eğer klasör kopyalanacak ise -r parametresi eklenir. Kısayollar $ . # sadece nokta bulunduğunuz dizini ifade eder $ ~/ # kullanıcının ana dizinini ifade eder $ history # yazmış olduğunuz komutların kaydını tutar ve bu komut ile erişebilirsiniz $ !\u0026lt;komut_sıralaması\u0026gt; # daha önce yazmış oldunuz komutu sıralamasının numarasını vererek calıştırabilirsiniz. $ yukarı(asagı)_okları # geçmiş komutlar arasında gezmenizi sağlar $ \u0026lt;tamamlanmamış_yol_veya_dosyaadı\u0026gt; TAB # Tab a bastığınızda sistem otomatik tamamlama işlemini gerçekleştirir. $ \u0026lt;tamamlanmamış komut\u0026gt; SHIFT\u0026amp;TAB # komutu otomatik tamamlar $ Ctrl a # imlecin en başa gitmesini sağlar $ Ctrl e # imlecin en sona gitmesini sağlar $ Ctrl d # imlec altındaki karakteri siler $ Ctrl k # imlecin bulunduğu sağlar $ Ctrl y # Ctrl k ile alınan içerik yapıştırılır. Yardım Alma $ man # genel yardım $ man wc # wc komutu hakkında yardım almanızı sağlar $ wc --help # wc komutu hakkında yardım almanızı sağlar $ info wc # wc komutu hakkında detaylı bilgi almanız sağlanır $ apropos wc # wc komutuna ait bütün yardım dosyaları güncellenir. Aradığınız Dosyayı Bulma Yöntemleri Arama yapmak $ find -name \u0026#34;*aramakistediğinizdesen*\u0026#34; # girdiğiniz desene göre bulunduğunuz dizinde arama yapmanızı sağlar. $ find /usr/local -name \u0026#34;*klas*\u0026#34; # isminin içerisinde klas geçen dosyaları ve klaksörleri listeler. $ find /usr/local -iname \u0026#34;*klas*\u0026#34; # yukarıdaki komutuna benzer şekilde, isminin içerisinde klas geçen dosyaları ve klasörleri listeler fakat bu durumda büyük veya küçük harfte olması dikkate alınmaz $ find ~ -type f -mtime -2 # 2 gün içerisinde değiştirilmiş bütün dosyaları listeler $ locate \u0026lt;aramakistediğinizdesen\u0026gt; # aradığınız dosyayı veya klasörü sistem genelinde arar. $ which \u0026lt;uygulama_adı\u0026gt; # uygulamanın nerede bulunduğunu gösterir $ whereis \u0026lt;uygulama_adı\u0026gt; # uygulamanın çalıştırılabilir dosyasının yerini gösterir $ dpkg -l | grep aramakistediğinizpaketismi # Debian paketleri içerisinde arama yaparak verilen desende bulunan paketleri listel Dosya içerisinde arama yapmak $ grep aranan_kelime dosya # dosya içerisinde aranan kelimenin nerelerde geçtiğini size aktarır $ grep -H aranan_kelime # -H çıktı dosyasını aranan kelimenin önüne koyar $ grep \u0026#39;aranan_kelime\u0026#39; dosya | wc # burada iki komut birleştirilmiştir, yani grep komutunun çıktısı wc komutuna girdi olmaktadir yani aranan kelime verilen dosyada 5 kere geçiyor ise bu durumda sonuc 5 olarak dönmektedir. $ find /home/kullanıcı_adı -name \u0026#39;*.txt\u0026#39; | xargs grep -c ^.* # verilen dizinde txt dosyalarını bularak bu dosyalar içerisindeki satır sayısını hesaplayarak çıktı vermektedir. İzinler \u0026amp; Hak Sahipliği $ ls -al # bu komut çalıştırıldıgında buna benzer bir çıktı görünebilir : drwxrwxrwx Burada bahsi geçen harflerin anlamları aşağıdaki gibi verilebilir.\nd: dizin rwx: oku, yaz, çalıştır ilk üçlü (rwx) : kullanıcı izinleri(u) ikinci üçlü: grup izinleri (g) üçüncü üçlü: diğer izinleri (o) ifade etmektedir. En başta d olursa bu o dosyanın aslında bir klasör olduğunu ifade etmektedir. Kullanıcı ve grupa, yazma ve çalıştırma izni vermek:\n$ chmod ug+rx dosya_ismi Kullanıcı haklarının alınması $ chmod ugo-rwx dosya_ismi \u0026#39;+\u0026#39; izin eklemeyi sağlar \u0026#39;-\u0026#39; izin silmenizi sağlar $ chmod +rx dosya_ismi/ VEYA $ chmod 755 dosya_ismi/ Hak sahipliğinin değiştirilmesi $ chown \u0026lt;kullanıcı_adı\u0026gt; \u0026lt;dosya veya klasör\u0026gt; # kullanıcı hak sahibini değiştirir $ chgrp \u0026lt;grup\u0026gt; \u0026lt;dosya veya klasör\u0026gt; # grup hak sahibini değiştirir $ chown \u0026lt;kullanıcı_adı\u0026gt;:\u0026lt;grup\u0026gt; \u0026lt;dosya veya klasör\u0026gt; # kullanıcı ve grup hak sahipliğini değiştirir Kullanışlı Linux Komutları $ df # sistem diskinin ne kadar dolu ve boş olduğu bilgisini gösterir $ free # ne kadar önbellek (RAM) alanının boş/dolu olduğunu gösterir $ uname -a # işletim sistemine ait temel bilgileri gösterir $ bc # terminal üzerinden hesap makinesi kullanmanızı sağlar $ /sbin/ifconfig # sunucunun ağ bilgilerini listeler. $ ln -s orjinal_dosyaismi yeni_dosyaismi # orjinal dosyaya link oluşturur $ du -sh # bulunduğunuz konumdaki disk kullanım bilgilerini listeler $ du -sh * # bulunduğunuz konumdaki dosyaların/klasörlerin kullanım bilgilerini gösterir $ du -s * | sort -nr # sıralanmış şekilde dosyaların/klasörlerin kullanımlarını listeler İşlem Yönetimi $ who # sisteme kimin girdiğini gösterir $ w # sistemde kimlerin olduğunu gösterir $ ps # arka planda çalışan işlemler hakkındaki bilgileri listeler. $ ps -e # sistemdeki bütün işlemleri listeler $ ps aux | grep \u0026lt;kullanıcı_adı\u0026gt; # kullanıcıya ait çalıştırılan işlemleri listeler $ top # CPU ve RAM değerlerinin kullanım bilgilerini gösterir $ mtop # birden fazla CPU için top komutunun yaptığını yapar $ Ctrl z \u0026lt;enter\u0026gt; bg or fg \u0026lt;enter\u0026gt; # çalışan işlemleri durdurur, arka plana atar (bg) veya ön plana getirir (fg) $ Ctrl c # yeni başlamış olan işlemi durdurur $ kill \u0026lt;işlem_no\u0026gt; # belirlenen işlemi sonlandırır, işlem ID sine göre belirlenir $ renice -n \u0026lt;önemlilik_değeri\u0026gt; # işlemin önemlilik değerini değiştirmenizi sağlar Text dosyalarını okumak $ less \u0026lt;dosya_ismi\u0026gt; # belirtilen dosyayı terminal üzerinden okumanızı sağlar G :dosyanın sonuna gider, g : dosyanın başına gider. $ more \u0026lt;dosya_ismi\u0026gt; # dosya içeriğini gösterir çıkmak için q ya basılması gereklidir. $ cat \u0026lt;dosya_ismi\u0026gt; #dosya içeriğini terminale yazdırır Metin Düzenleyicileri VI ve VIM Terminal tabanlı güçlü metin düzenleyicidir. Vi genelde linux tabanlı bütün sistemlerde mevcuttur, vim, vi nin gelişmişidir.\nEMACS Grafik tabanlı metin düzenleyicidir. Bu düzenleyiciye ait olan klavye düzeni hakkında bilginiz olmalıdır. Bütün linux ve unix tabanlı sistemlerde mevcuttur.\nXEMACS EMACS in çok daha gelişmişidir, yazım hataları, web ve metini iyi bir şekilde kontrol etmek mümkündür fakat normalde yüklenmiş olmaz.\nPICO Terminal tabanlı basit metin düzenleyicidir, buna ait klavye düzeni bilinmelidir.\nVIM Temelleri Temeller $ vim dosya_ismi # dosya_ismin de dosya oluşturur veya yazma modunda açar $ i # vim \u0026#39;in içerisine girdikten sonra i açık olan dosyaya bir şeyler yazmanıza olanak sağlar. $ ESC # dosya düzenleme modundan çıkılır $ : # vim içerisinde kullanacağınız komutlar : ile başlar $ :w # vim içerisinde :w yaptığınızda yazdığınızı kaydeder. $ :q # bu komut vim den çıkmanızı sağlar $ :q! # hiç birşeyi kaydetmeden çıkmanızı sağlar. $ :wq # kaydederek çıkar $ R # vim içerisinde özellik değiştirmenizi sağlar $ r # imlecin bulunduğu karakteri değiştirmenizi sağlar $ q: # vim içerisinde yazdığınız komutların kaydını gösterir $ :w yeni_dosyaadı # yeni dosyaya kaydeder. $ :#,#w yeni_dosyaadı# belirlenen (#,#) aralıktaki metini yeni dosyaya kaydeder. $ :# belirlenen (#) satıra gitmenizi sağlar Yardım $ vimtutor # vim içerisinde nasıl çalıştığına dair bilgileri içeren tur başlatılır. $ :help # vim içerisinde yardım açar, çıkmak için q komutu kullanılır. $ :help \u0026lt;konu\u0026gt; # belirlenen konu hakkında yardım açar $ :help \u0026lt;konu\u0026gt; CTRL-D # belirlenen konunun geçtiği bütün yardım dökümanını listeler. $ :\u0026lt;yukarı-asagı tusları\u0026gt; # daha önceki yaptığınız komutlar arasında gezmenizi sağlar. Dosya içerisinde gezme (vim içerisinde) $ $ # bulunduğunuz satırın en sonuna gider $ A # bulunduğunuz satırın en sonuna yazma modunu açarak gider $ 0 (sıfır) # satırın başlangıcına gider $ CTRL-g # imlecin nerede olduğu ve o satır hakkında bilgi verir $ SHIFT-G # imleci dosyanın en sonuna getirir Görüntü (vim içerisinde) WRAPPING AND LINE NUMBERS $ :set nowrap # kelimelerin kaymamalarını sağlar $ :set number # satır numaralarını gösterir Arşivleme ve Sıkıştırma $ tar -cvf dosya_ismi.tar klasör/ # verilen klasör için arşiv oluşturur $ tar -czvf dosya_ismi.tgz klasör/ # verilen klasör için arşivlenmiş ve sıkıştırılmış dosya oluşturur. Arşivleri görüntüleme $ tar -tvf dosya_ismi.tar $ tar -tzvf dosya_ismi.tgz Çıkartma $ tar -xvf dosya_ismi.tar $ tar -xzvf dosya_ismi.tgz $ gunzip dosya_ismi.tar.gz $ tar zxf blast.linux.tar.Zs Basit yükleme işlemleri RPM yüklemeleri $ rpm -i uygulama_ismi.rpm $ rpm --query \u0026lt;paket_ismi\u0026gt; ## RPM versiyonunu kontrol etme için Debian paketlerinin yüklenmesi $ apt-cache search nmap # nmap adındaki uygulamayı debian deposundan arama yapar $ apt-cache show nmap # nmap hakkında tanımlamayı(bilgi) gösterir $ apt-get install nmap # nmap \u0026#39;i sisteme kurar. $ apt-get update # sistemdeki uygulamaları ve servisleri günceller $ apt-get upgrade -u # uygulamaların yeni versiyonu var ise yükseltme yapar $ dpkg -i dosya.deb # indirilen debian dosyasının yüklenmesini sağlar $ aptitude # apt-get ile aynı işlemi görür $ aptitude search vim # vim programını debian deposunda arar Cihazlar Takma /Çıkarma usb/floppy/cdrom $ mount /media/usb $ umount /media/usb $ mount /media/cdrom $ eject /media/cdrom $ mount /media/floppy Çevresel Değişkenler $ xhost user@host # kullanıcı için çalıştırma izini ekler $ echo DISPLAY # ekranın ayarlarını gösterir $ export (setenv) DISPLAY=\u0026lt;lokal_IP\u0026gt;:0 # görüntü değişkeninin değerini değiştirir $ unsetenv DISPLAY # görüntü değişkenini siler $ printenv # kullanılan çevresel değişkenleri listeler $ $PATH # terminal üzerinde programların çalışmasını sağlayan yolları gösterir. ","permalink":"https://mrturkmen.com/posts/linux-temeller-1/","summary":"\u003ch2 id=\"özet\"\u003eÖzet:\u003c/h2\u003e\n\u003cp\u003eBu yazıda linux ortamına biraz daha giriş yaparak, linux ortamında bulunan komutlar hakkında kısa bilgilendirme yapılması planlanmaktadır.\u003c/p\u003e\n\u003ch2 id=\"giriş\"\u003eGiriş\u003c/h2\u003e\n\u003ch4 id=\"neden-linux-\"\u003eNeden Linux ?\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eBirden fazla işlemi aynı anda kolay şekilde yapmanızı sağlar\u003c/li\u003e\n\u003cli\u003eUzaktan işlemlerinizi halletmede büyük kolaylık sağlar\u003c/li\u003e\n\u003cli\u003eBirden fazla kullanıcı aynı sunucuya erişebilir\u003c/li\u003e\n\u003cli\u003eTerminale, bir sistem üzerinde olan kaynaklara birden fazla erişim mümkündür\u003c/li\u003e\n\u003cli\u003eArayüz olan sistemlere göre daha performanslı, Bedava , Güncel\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"temeller\"\u003eTemeller\u003c/h2\u003e\n\u003ch4 id=\"bu-bilgilendirme-dosyası-için-not\"\u003eBu bilgilendirme dosyası için not\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eBütün komutlar büyük ve küçük harfe duyarlıdır.\u003c/strong\u003e\u003c/p\u003e","title":"debian: terminal/komut "},{"content":"Özet: Bu kısa yazımızda linux bilgisayarlarının terminali üzerinden yapabileceğiniz basit işlemlere dair bilgiler verilecektir.\nLinux tabanlı sunucularda/bilgisayarda terminal üzerinden kopyalama Kopyalama işlemi \u0026ldquo;cp\u0026rdquo; komutu ile yapılmaktadır, bu komuta ait format aşağıdaki gibi özetlenebilir.\ncp [parametreler] [kopyalanacak-dosya] [kopyalanmasi-hedeflenen-yer] Bu komutun kullanımına örnek verelim, kopyalanacak dizin ve kopyalanması gereken dosya ;\nKopyalanacak dizin : /home/geek/Masaustu/\nKopyalanacak dosya : /home/geek/Dokumanlar/resim.png\nBu durumda komut : (* Dizinlere ve dosyalara erişim hakkına sahip olduğunuzdan emin olunuz)\ncp /home/geek/Dokumanlar/resim.png /home/geek/Masaustu/ Eğer bir dizin içerisindeki bütün dosyalar kopyalanması planlanıyor ise -R parametresi kullanılması gerekmektedir. Diyelim ki bir dizin içerisinde 1.png, 2.png, 3.png \u0026hellip; 12.png gibi dosyalar var ise ve bu dizinin adı \u0026ldquo;resimler\u0026rdquo; ise, resimler dizisi istenilen diziye aşağıdaki komut yardımı ile kopyalanabilir.\ncp -R /home/geek/Dokumanlar/resimler /home/geek/Masaustu/ Bu kısımda bütün veri hedeflenen dizine aktarılmaktadır.\ncp komutuna ait bazı parametreler ve onların kısa açıklamaları:\n-R : verilen dizindeki bütün dosyaları hedeflenen dizine kopyalamak için gereklidir.\n-p : kopyalama yaparken dosyaya ait olan, oluşturulma, değiştirme, sahiplik bilgilerini değiştirmeden onlar ile birlikte kopyalamak için gereklidir.\nTerminal üzerinden yeniden başlatma Linux terminali üzerinden bir sunucuyu yeniden başlatmak veya kapatmak için gerekli olan komutlar şu şekilde özetlenebilir. Bu komutların çalışması için sunucu üzerinde yönetici (root) yetkilerine sahip olmalısınız.\nYeniden başlatmak için\nreboot Bu kısımda \u0026quot;Permission denied\u0026quot; veya buna benzer bir izin reddedildi mesajı aldığınızda aynı komutu sudo eki koyarak denemelisiniz, yönetici yetkisi olmadığı durumda bu tür mesajları alırsınız.\nsudo reboot Bu komut bazen her linux dağılımı için geçerli olmayabilir bu durumda aşağıda verilen komut reboot komutuna alternatif olarak verilebilir. (* Bu komut MacOS bilgisayarlar içinde kullanılabilir, MacOS işletim sistemleri Hybrid bir yapıya sahip olduğundan dolayı Unix çekirdeği içermektedir.)\nshutdown -r now \u0026quot;shutdown\u0026quot; komutunda isterseniz bekleme süresi ekleyebilirsiniz böylece sunucu veya bilgisayar, belirlenen bekleme süresi sonunda verdiğiniz komutu uygulayacaktır.\nshutdown -r +30 30 dk sonrasında sunucu yeniden başlatılacaktır, benzer şekilde dk belirlemektense, istediğiniz saatte bu işlemi yapmak isterseniz, istediğiniz saati aşağıdaki formatta ayarlanabilir.\nshutdown -r 19:30 Sunucu, saat 19:30 da yeniden başlatılmaya ayarlanmıştır.\n\u0026quot;shutdown\u0026quot; komutuna ait bazı parametreler ve açıklamaları şu şekilde özetlenebilir.\n-r : sunucuya yeniden başlatma sinyalini vermesi için gereklidir.\n-k : sunucuya bağlı olan fakat yönetici yetkisinde erişmemiş kişileri sunucudan atar.\n","permalink":"https://mrturkmen.com/posts/linux-terminalinden-basit-komutlar/","summary":"\u003ch2 id=\"özet\"\u003eÖzet:\u003c/h2\u003e\n\u003cp\u003eBu kısa yazımızda linux bilgisayarlarının terminali üzerinden yapabileceğiniz basit işlemlere dair bilgiler verilecektir.\u003c/p\u003e\n\u003ch4 id=\"linux-tabanlı-sunuculardabilgisayarda-terminal-üzerinden-kopyalama\"\u003eLinux tabanlı sunucularda/bilgisayarda terminal üzerinden kopyalama\u003c/h4\u003e\n\u003cp\u003eKopyalama işlemi \u0026ldquo;cp\u0026rdquo; komutu ile yapılmaktadır, bu komuta ait format aşağıdaki gibi özetlenebilir.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e cp \u003cspan style=\"color:#f92672\"\u003e[\u003c/span\u003eparametreler\u003cspan style=\"color:#f92672\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e[\u003c/span\u003ekopyalanacak-dosya\u003cspan style=\"color:#f92672\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e[\u003c/span\u003ekopyalanmasi-hedeflenen-yer\u003cspan style=\"color:#f92672\"\u003e]\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eBu komutun kullanımına örnek verelim,  kopyalanacak dizin ve kopyalanması gereken dosya ;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKopyalanacak dizin\u003c/strong\u003e : \u003ccode\u003e/home/geek/Masaustu/\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKopyalanacak dosya\u003c/strong\u003e : \u003ccode\u003e/home/geek/Dokumanlar/resim.png\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eBu durumda komut :  (* Dizinlere ve dosyalara erişim hakkına sahip olduğunuzdan emin olunuz)\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ecp /home/geek/Dokumanlar/resim.png /home/geek/Masaustu/\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eEğer bir dizin içerisindeki bütün dosyalar kopyalanması planlanıyor ise -R parametresi kullanılması gerekmektedir. Diyelim ki bir dizin içerisinde 1.png, 2.png, 3.png \u0026hellip; 12.png gibi dosyalar var ise ve bu dizinin adı \u0026ldquo;resimler\u0026rdquo; ise, resimler dizisi istenilen diziye aşağıdaki komut yardımı ile kopyalanabilir.\u003c/p\u003e","title":"debian: cp/reboot komuları "}]