roomservice: Add lightweight dependencies to repositories
Roomservice can already fetch your cm_<device> without the need for a
manifest entry.
However, when working with common repositories, there is no way of
actually fetching them without adding to the manifest. This patch
introduces a lightweight dependency system. Each repository can have a
cm.dependencies in the following json format:
[
{
"repository": "repository_name_on_cm_organization"
"target_path": "target/path"
},
...
]
For instance, for cm_anzu I need android_device_semc_msm7x30-common and
android_device_semc_mogami-common. I would add both to cm.dependencies
as follows:
[
{
"repository": "android_device_semc_msm7x30-common",
"target_path": "device/semc/msm7x30-common"
},
{
"repository": "android_device_semc_mogami-common",
"target_path": "device/semc/mogami-common"
}
]
Roomservice would then fetch the anzu repository, parse the dependency
files and add/fetch/sync these additional repositories if they don't
exist already.
This also adds pretty printing to the output xml.
Change-Id: I9cc847adfc717a06439bc6094213ed6492343158
roomservice: Add branch support to cm.dependencies
Allow the cm.dependencies entries to provide an optional "branch" for
the repository dependencies. Added to fully support
http://wiki.cyanogenmod.com/wiki/Integrated_kernel_building
Change-Id: I35b51920d296afa329411af6172c7bd9aeef4af8
roomservice: Fill in dependencies for already-deployed repositories
Change-Id: I01fd408c9c4bfa78097c7f848b2556d2b2b180f3
roomservice: Extend dependency-checks to devices in main manifest
CM currently keeps devices inherited from AOSP in the main manifest,
so take that into account as well when checking device paths
Change-Id: I9663f283617f237428b4eaa0cd60b5de2b86a7b9
make compatible with github v3 api
Change-Id: Iff6f1f9099cdc5d2b49e04000b5fe3d04aa5d7e4
Fixed build for full-eng
Previously
Traceback (most recent call last):
File "build/tools/roomservice.py", line 153, in <module>
repo_path = get_from_manifest(device)
NameError: name 'device' is not defined
** Don't have a product spec for: 'full'
** Do you have the right repo manifest?
Now
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=4.0.4
TARGET_PRODUCT=full
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
HOST_ARCH=x86
HOST_OS=linux
HOST_BUILD_TYPE=release
BUILD_ID=IMM76L
============================================
Change-Id: Ib513705aba9a7a52a971ab64102ecbe9fddfb97a
roomservice: Bump github request per_page to 100
Change the number of repos per page from the default 30
to 100.
We seem to be hitting the rate limit on the jenkins server.
Change-Id: Ie733feaa0414cbfebb7efcfc1e24d94e1e466d1b
roomservice: Add support for netrc
Change-Id: I1f5e11e40125abd0c4e4c8d8294d4fc09bfdc30a
roomservice: Handle missing netrc file
Change-Id: If981fe79dc3e2191434301239b0cd585be8b4730
roomservice.py: Verbose error when a branch is missing.
Also add ROOMSERVICE_BRANCHES environment variable to use fallback branches.
Change-Id: I3c2b1d79fc185c1f1e1d658e5ca4f78e688780e2
roomservice.py: Fixups around fallback branches not being used by dependencies when ROOMSERVICE_BRANCHES is defined.
Change-Id: Ifb42a023cae5f62ac8f9cf7832125b91b431169c
roomservice: Allow following up tag references
This is now needed for release builds
Change-Id: I8c5f87341059b3b15ee853312b71df73790ad0d8
build: local_manifest.xml deprecated, use local_manifests
Patch Set 2:- Use roomservice.xml instead of cm.xml.
Change-Id: I3d8a6ef3907b92808662cbba912cea5ed38d0bde
Fix fallback branch search in roomservice
If you provided a fallback branch to roomservice via the
ROOMSERVICE_BRANCHES environment var the branch search would fail
if the device repo had any tags.
Fixed this by appending the tag search results to the branch
search results instead of overwriting them
Change-Id: I73a11af1500bd04e346f08ec3f83454502f3a169
roomservice: Fix wrong assumption about path of active manifest
Change-Id: Id740ff4b848e6ccbfd658be4846197b8ca519237
roomservice: When validating the presence of a repo, check main manifest too
Change-Id: If680536484074b473458723d93e783d074d7f669
roomservice: Bump devices per page to 200
Limit was reached again when attempting to
lunch various projects, such as steelhead.
Change-Id: I2f3b9705e07e1e47b86857aeb383cf7c99fcdbdc
Signed-off-by: William Roberts <bill.c.roberts@gmail.com>
roomservice: Fix assumptions about the branch naming
We can't just split from the last slash anymore, since we're using
those to distinguish the stabilization branches
Change-Id: Ia175dd317f508e99b275b56e9c83bd4729a75ddb
roomservice: Add recursive dependencies
Dependency repositories can now have dependencies themselves
Change-Id: I33a28709170da52bc98f4a62387927e3a11b2450
diff --git a/tools/roomservice.py b/tools/roomservice.py
index 61f8555..f5aac06 100755
--- a/tools/roomservice.py
+++ b/tools/roomservice.py
@@ -1,55 +1,275 @@
#!/usr/bin/env python
+# Copyright (C) 2012-2013, The CyanogenMod Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
import os
import sys
import urllib2
import json
+import re
+import netrc, base64
from xml.etree import ElementTree
product = sys.argv[1];
-device = product[product.index("_") + 1:]
-print "Device %s not found. Attempting to retrieve device repository from CyanogenMod Github (http://github.com/CyanogenMod)." % device
+
+if len(sys.argv) > 2:
+ depsonly = sys.argv[2]
+else:
+ depsonly = None
+
+try:
+ device = product[product.index("_") + 1:]
+except:
+ device = product
+
+if not depsonly:
+ print "Device %s not found. Attempting to retrieve device repository from CyanogenMod Github (http://github.com/CyanogenMod)." % device
repositories = []
+try:
+ authtuple = netrc.netrc().authenticators("api.github.com")
+
+ if authtuple:
+ githubauth = base64.encodestring('%s:%s' % (authtuple[0], authtuple[2])).replace('\n', '')
+ else:
+ githubauth = None
+except:
+ githubauth = None
+
+def add_auth(githubreq):
+ if githubauth:
+ githubreq.add_header("Authorization","Basic %s" % githubauth)
+
page = 1
-while True:
- result = json.loads(urllib2.urlopen("http://github.com/api/v2/json/repos/show/CyanogenMod?page=%d" % page).read())
- if len(result['repositories']) == 0:
+while not depsonly:
+ githubreq = urllib2.Request("https://api.github.com/users/CyanogenMod/repos?per_page=200&page=%d" % page)
+ add_auth(githubreq)
+ result = json.loads(urllib2.urlopen(githubreq).read())
+ if len(result) == 0:
break
- repositories = repositories + result['repositories']
+ for res in result:
+ repositories.append(res)
page = page + 1
-for repository in repositories:
- repo_name = repository['name']
- if repo_name.startswith("android_device_") and repo_name.endswith("_" + device):
- print "Found repository: %s" % repository['name']
- manufacturer = repo_name.replace("android_device_", "").replace("_" + device, "")
-
- try:
- lm = ElementTree.parse(".repo/local_manifest.xml")
- lm = lm.getroot()
- except:
- lm = ElementTree.Element("manifest")
-
- for child in lm.getchildren():
- if child.attrib['name'].endswith("_" + device):
- print "Duplicate device '%s' found in local_manifest.xml." % child.attrib['name']
- sys.exit()
+local_manifests = r'.repo/local_manifests'
+if not os.path.exists(local_manifests): os.makedirs(local_manifests)
- repo_path = "device/%s/%s" % (manufacturer, device)
- project = ElementTree.Element("project", attrib = { "path": repo_path, "remote": "github", "name": "CyanogenMod/%s" % repository['name'] })
+def exists_in_tree(lm, repository):
+ for child in lm.getchildren():
+ if child.attrib['name'].endswith(repository):
+ return True
+ return False
+
+# in-place prettyprint formatter
+def indent(elem, level=0):
+ i = "\n" + level*" "
+ if len(elem):
+ if not elem.text or not elem.text.strip():
+ elem.text = i + " "
+ if not elem.tail or not elem.tail.strip():
+ elem.tail = i
+ for elem in elem:
+ indent(elem, level+1)
+ if not elem.tail or not elem.tail.strip():
+ elem.tail = i
+ else:
+ if level and (not elem.tail or not elem.tail.strip()):
+ elem.tail = i
+
+def get_default_revision():
+ m = ElementTree.parse(".repo/manifest.xml")
+ d = m.findall('default')[0]
+ r = d.get('revision')
+ return r.replace('refs/heads/', '').replace('refs/tags/', '')
+
+def get_from_manifest(devicename):
+ try:
+ lm = ElementTree.parse(".repo/local_manifests/roomservice.xml")
+ lm = lm.getroot()
+ except:
+ lm = ElementTree.Element("manifest")
+
+ for localpath in lm.findall("project"):
+ if re.search("android_device_.*_%s$" % device, localpath.get("name")):
+ return localpath.get("path")
+
+ # Devices originally from AOSP are in the main manifest...
+ try:
+ mm = ElementTree.parse(".repo/manifest.xml")
+ mm = mm.getroot()
+ except:
+ mm = ElementTree.Element("manifest")
+
+ for localpath in mm.findall("project"):
+ if re.search("android_device_.*_%s$" % device, localpath.get("name")):
+ return localpath.get("path")
+
+ return None
+
+def is_in_manifest(projectname):
+ try:
+ lm = ElementTree.parse(".repo/local_manifests/roomservice.xml")
+ lm = lm.getroot()
+ except:
+ lm = ElementTree.Element("manifest")
+
+ for localpath in lm.findall("project"):
+ if localpath.get("name") == projectname:
+ return 1
+
+ ## Search in main manifest, too
+ try:
+ lm = ElementTree.parse(".repo/manifest.xml")
+ lm = lm.getroot()
+ except:
+ lm = ElementTree.Element("manifest")
+
+ for localpath in lm.findall("project"):
+ if localpath.get("name") == projectname:
+ return 1
+
+ return None
+
+def add_to_manifest(repositories, fallback_branch = None):
+ try:
+ lm = ElementTree.parse(".repo/local_manifests/roomservice.xml")
+ lm = lm.getroot()
+ except:
+ lm = ElementTree.Element("manifest")
+
+ for repository in repositories:
+ repo_name = repository['repository']
+ repo_target = repository['target_path']
+ if exists_in_tree(lm, repo_name):
+ print 'CyanogenMod/%s already exists' % (repo_name)
+ continue
+
+ print 'Adding dependency: CyanogenMod/%s -> %s' % (repo_name, repo_target)
+ project = ElementTree.Element("project", attrib = { "path": repo_target,
+ "remote": "github", "name": "CyanogenMod/%s" % repo_name })
+
+ if 'branch' in repository:
+ project.set('revision',repository['branch'])
+ elif fallback_branch:
+ print "Using fallback branch %s for %s" % (fallback_branch, repo_name)
+ project.set('revision', fallback_branch)
+ else:
+ print "Using default branch for %s" % repo_name
+
lm.append(project)
-
- raw_xml = ElementTree.tostring(lm)
- raw_xml = '<?xml version="1.0" encoding="UTF-8"?>\n' + raw_xml
- f = open('.repo/local_manifest.xml', 'w')
- f.write(raw_xml)
- f.close()
-
- print "Syncing repository to retrieve project."
- os.system('repo sync %s' % repo_path)
- print "Done!"
- sys.exit()
+ indent(lm, 0)
+ raw_xml = ElementTree.tostring(lm)
+ raw_xml = '<?xml version="1.0" encoding="UTF-8"?>\n' + raw_xml
-print "Repository for %s not found in the CyanogenMod Github repository list. If this is in error, you may need to manually add it to your local_manifest.xml." % device
+ f = open('.repo/local_manifests/roomservice.xml', 'w')
+ f.write(raw_xml)
+ f.close()
+
+def fetch_dependencies(repo_path, fallback_branch = None):
+ print 'Looking for dependencies'
+ dependencies_path = repo_path + '/cm.dependencies'
+ syncable_repos = []
+
+ if os.path.exists(dependencies_path):
+ dependencies_file = open(dependencies_path, 'r')
+ dependencies = json.loads(dependencies_file.read())
+ fetch_list = []
+
+ for dependency in dependencies:
+ if not is_in_manifest("CyanogenMod/%s" % dependency['repository']):
+ fetch_list.append(dependency)
+ syncable_repos.append(dependency['target_path'])
+
+ dependencies_file.close()
+
+ if len(fetch_list) > 0:
+ print 'Adding dependencies to manifest'
+ add_to_manifest(fetch_list, fallback_branch)
+ else:
+ print 'Dependencies file not found, bailing out.'
+
+ if len(syncable_repos) > 0:
+ print 'Syncing dependencies'
+ os.system('repo sync %s' % ' '.join(syncable_repos))
+
+ for deprepo in syncable_repos:
+ fetch_dependencies(deprepo)
+
+def has_branch(branches, revision):
+ return revision in [branch['name'] for branch in branches]
+
+if depsonly:
+ repo_path = get_from_manifest(device)
+ if repo_path:
+ fetch_dependencies(repo_path)
+ else:
+ print "Trying dependencies-only mode on a non-existing device tree?"
+
+ sys.exit()
+
+else:
+ for repository in repositories:
+ repo_name = repository['name']
+ if repo_name.startswith("android_device_") and repo_name.endswith("_" + device):
+ print "Found repository: %s" % repository['name']
+
+ manufacturer = repo_name.replace("android_device_", "").replace("_" + device, "")
+
+ default_revision = get_default_revision()
+ print "Default revision: %s" % default_revision
+ print "Checking branch info"
+ githubreq = urllib2.Request(repository['branches_url'].replace('{/branch}', ''))
+ add_auth(githubreq)
+ result = json.loads(urllib2.urlopen(githubreq).read())
+
+ ## Try tags, too, since that's what releases use
+ if not has_branch(result, default_revision):
+ githubreq = urllib2.Request(repository['tags_url'].replace('{/tag}', ''))
+ add_auth(githubreq)
+ result.extend (json.loads(urllib2.urlopen(githubreq).read()))
+
+ repo_path = "device/%s/%s" % (manufacturer, device)
+ adding = {'repository':repo_name,'target_path':repo_path}
+
+ fallback_branch = None
+ if not has_branch(result, default_revision):
+ if os.getenv('ROOMSERVICE_BRANCHES'):
+ fallbacks = filter(bool, os.getenv('ROOMSERVICE_BRANCHES').split(' '))
+ for fallback in fallbacks:
+ if has_branch(result, fallback):
+ print "Using fallback branch: %s" % fallback
+ fallback_branch = fallback
+ break
+
+ if not fallback_branch:
+ print "Default revision %s not found in %s. Bailing." % (default_revision, repo_name)
+ print "Branches found:"
+ for branch in [branch['name'] for branch in result]:
+ print branch
+ print "Use the ROOMSERVICE_BRANCHES environment variable to specify a list of fallback branches."
+ sys.exit()
+
+ add_to_manifest([adding], fallback_branch)
+
+ print "Syncing repository to retrieve project."
+ os.system('repo sync %s' % repo_path)
+ print "Repository synced!"
+
+ fetch_dependencies(repo_path, fallback_branch)
+ print "Done"
+ sys.exit()
+
+print "Repository for %s not found in the CyanogenMod Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml." % device